feat(files): 大文件分片上传、停服解压 zip 先列冲突再覆盖、文件和文件夹可下载;导出和下载不再带出 RCON 密码与转发密钥

This commit is contained in:
Lemon-miaow committed 2026-09-28 22:44:03 +08:00
1 parent 3abf146321
commit 1d1549cea2
76 files changed
+10542 -569

No files matched your search

+215
View File
@@ -0,0 +1,215 @@
package fileedit
import (
"archive/zip"
"bytes"
"compress/flate"
"context"
"errors"
"fmt"
"io"
"io/fs"
"os"
"path"
)
// Content types of a download: a file goes out as its bytes, a folder as a zip.
const (
DownloadFileType = "application/octet-stream"
DownloadZipType = "application/zip"
)
// Download is one file or folder of a world on its way to the owner's browser,
// run by the export Job (cmd/felis export --mode files). It passes the same
// guards a read does (Guard): the forwarding-secret file never leaves, and
// server.properties leaves with its RCON password redacted.
type Download struct {
// Size is a file download's exact length, or -1 for a folder, whose zip is
// written as it streams.
Size int64
ContentType string
// Skipped and Withheld count, once WriteTo has run, the entries a folder
// download left out: links, devices and sockets, and guarded files.
Skipped, Withheld int
root *os.Root
name string
file *os.File // a file download
body []byte // a redacted file download
guard Guard
}
// OpenDownload opens name under rootPath for download. dir is what the caller
// saw at name when it asked (the panel's listing): a download of a file that has
// since become a folder, or the reverse, is refused rather than sent as the
// other thing. The world root itself is refused; the world export sends that.
func OpenDownload(rootPath, name string, dir bool) (*Download, error) {
name = path.Clean(name)
if name == "." || name == "/" || !fs.ValidPath(name) {
return nil, fmt.Errorf("%s is not a file or folder inside the world", name)
}
r, err := os.OpenRoot(rootPath)
if err != nil {
return nil, fmt.Errorf("open the world root: %w", err)
}
d := &Download{root: r, name: name, guard: NewGuard(r)}
if err := d.open(dir); err != nil {
r.Close()
return nil, err
}
return d, nil
}
func (d *Download) open(dir bool) error {
info, err := d.root.Stat(d.name)
if err != nil {
return err
}
if info.IsDir() != dir {
if info.IsDir() {
return fmt.Errorf("%s is a folder now; reload the file list and download it again", d.name)
}
return fmt.Errorf("%s is not a folder now; reload the file list and download it again", d.name)
}
if dir {
d.Size, d.ContentType = -1, DownloadZipType
return nil
}
if !info.Mode().IsRegular() {
return fmt.Errorf("%s is not a regular file", d.name)
}
withhold, redact := d.guard.Rule(info)
if withhold {
return fmt.Errorf("%s is the file holding the proxy forwarding secret, which is shared cluster-wide, and cannot be downloaded", d.name)
}
f, err := d.root.Open(d.name)
if err != nil {
return err
}
d.ContentType = DownloadFileType
if !redact {
d.file, d.Size = f, info.Size()
return nil
}
defer f.Close()
b, err := io.ReadAll(io.LimitReader(f, MaxReadBytes+1))
if err != nil {
return err
}
if len(b) > MaxReadBytes {
return fmt.Errorf("%s is over %d bytes and cannot be redacted for download", d.name, MaxReadBytes)
}
d.body = RedactProps(b)
d.Size = int64(len(d.body))
return nil
}
// Close releases what OpenDownload opened.
func (d *Download) Close() error {
if d.file != nil {
d.file.Close()
}
return d.root.Close()
}
// WriteTo writes the download to w: exactly Size bytes of a file, or a zip of a
// folder whose entries sit under the folder's own name, so unpacking it makes
// that one folder. A file that shrank since it was opened is an error, never a
// short download passed off as whole.
func (d *Download) WriteTo(ctx context.Context, w io.Writer) error {
switch {
case d.body != nil:
_, err := w.Write(d.body)
return err
case d.file != nil:
_, err := io.CopyN(w, ctxReader{ctx, d.file}, d.Size)
if errors.Is(err, io.EOF) {
return fmt.Errorf("%s shrank while it was being downloaded", d.name)
}
return err
}
return d.writeZip(ctx, w)
}
// writeZip streams the folder as a zip. Everything is deflated at the fastest
// level: the Job has one CPU and the owner's connection is the slower end, and
// already-compressed files (jars, region files) come out as stored blocks
// without costing much. Links, devices and sockets are left out, like a world
// export leaves them out.
func (d *Download) writeZip(ctx context.Context, w io.Writer) error {
zw := zip.NewWriter(w)
zw.RegisterCompressor(zip.Deflate, func(out io.Writer) (io.WriteCloser, error) {
return flate.NewWriter(out, flate.BestSpeed)
})
base := path.Base(d.name)
err := fs.WalkDir(d.root.FS(), d.name, func(p string, de fs.DirEntry, err error) error {
if err != nil {
return err
}
if err := ctx.Err(); err != nil {
return err
}
entry := base + p[len(d.name):]
info, err := de.Info()
if err != nil {
return err
}
switch {
case de.IsDir():
hdr := &zip.FileHeader{Name: entry + "/", Modified: info.ModTime()}
hdr.SetMode(info.Mode().Perm() | fs.ModeDir)
_, err := zw.CreateHeader(hdr)
return err
case !de.Type().IsRegular():
d.Skipped++
return nil
}
withhold, redact := d.guard.Rule(info)
if withhold {
d.Withheld++
return nil
}
f, err := d.root.Open(p)
if err != nil {
return err
}
defer f.Close()
var src io.Reader = ctxReader{ctx, f}
if redact {
b, err := io.ReadAll(io.LimitReader(f, MaxReadBytes+1))
if err != nil {
return err
}
if len(b) > MaxReadBytes {
d.Withheld++
return nil
}
src = bytes.NewReader(RedactProps(b))
}
hdr := &zip.FileHeader{Name: entry, Method: zip.Deflate, Modified: info.ModTime()}
hdr.SetMode(info.Mode().Perm())
fw, err := zw.CreateHeader(hdr)
if err != nil {
return err
}
_, err = io.Copy(fw, src)
return err
})
if err != nil {
return err
}
return zw.Close()
}
// ctxReader stops a long copy once ctx is done.
type ctxReader struct {
ctx context.Context
r io.Reader
}
func (c ctxReader) Read(p []byte) (int, error) {
if err := c.ctx.Err(); err != nil {
return 0, err
}
return c.r.Read(p)
}
+304
View File
@@ -0,0 +1,304 @@
package fileedit
import (
"archive/zip"
"bytes"
"context"
"fmt"
"io"
"os"
"path/filepath"
"reflect"
"strings"
"syscall"
"testing"
)
// downloadWorld is worldRoot plus the guarded files with real secrets in them.
func downloadWorld(t *testing.T) string {
t.Helper()
root, _ := worldRoot(t)
for name, body := range map[string]string{
"server.properties": "motd=hi\nrcon.password=hunter2\n",
"config/paper-global.yml": "secret: aVeryRealForwardingKey\n",
} {
if err := os.WriteFile(filepath.Join(root, name), []byte(body), 0o644); err != nil {
t.Fatal(err)
}
}
return root
}
// download runs a whole download into memory.
func download(t *testing.T, root, name string, dir bool) (*Download, []byte) {
t.Helper()
d, err := OpenDownload(root, name, dir)
if err != nil {
t.Fatalf("OpenDownload(%s): %v", name, err)
}
defer d.Close()
var out bytes.Buffer
if err := d.WriteTo(context.Background(), &out); err != nil {
t.Fatalf("WriteTo(%s): %v", name, err)
}
return d, out.Bytes()
}
// unzipped reads a zip into name → content ("<dir> <mode>" for folders).
func unzipped(t *testing.T, b []byte) map[string]string {
t.Helper()
zr, err := zip.NewReader(bytes.NewReader(b), int64(len(b)))
if err != nil {
t.Fatal(err)
}
got := map[string]string{}
for _, f := range zr.File {
if f.FileInfo().IsDir() {
// The Unix mode, S_IFDIR included, which is what unzip tools
// restore a folder's permissions from.
got[f.Name] = fmt.Sprintf("<dir %o>", f.ExternalAttrs>>16)
continue
}
rc, err := f.Open()
if err != nil {
t.Fatal(err)
}
body, err := io.ReadAll(rc)
rc.Close()
if err != nil {
t.Fatal(err)
}
got[f.Name] = f.Mode().Perm().String() + " " + string(body)
}
return got
}
func TestDownload(t *testing.T) {
const redacted = "motd=hi\nrcon.password=" + redactedValue + "\n"
t.Run("a file goes out as its exact bytes", func(t *testing.T) {
root := downloadWorld(t)
d, got := download(t, root, "config/paper.yml", false)
if string(got) != "verbose: false\n" || d.Size != int64(len(got)) || d.ContentType != DownloadFileType {
t.Fatalf("download = %q, size %d, type %s", got, d.Size, d.ContentType)
}
})
t.Run("server.properties, under any name, goes out redacted", func(t *testing.T) {
root := downloadWorld(t)
if err := os.Link(filepath.Join(root, "server.properties"), filepath.Join(root, "copy.txt")); err != nil {
t.Fatal(err)
}
symlink(t, "server.properties", filepath.Join(root, "sym.txt"))
for _, name := range []string{"server.properties", "./server.properties", "copy.txt", "sym.txt"} {
d, got := download(t, root, name, false)
if string(got) != redacted || d.Size != int64(len(redacted)) {
t.Errorf("%s: download = %q, size %d; want %q", name, got, d.Size, redacted)
}
}
})
t.Run("the forwarding secret, under any name, is refused", func(t *testing.T) {
root := downloadWorld(t)
if err := os.Link(filepath.Join(root, "config/paper-global.yml"), filepath.Join(root, "hard.yml")); err != nil {
t.Fatal(err)
}
symlink(t, "config", filepath.Join(root, "cfg"))
for _, name := range []string{"config/paper-global.yml", "hard.yml", "cfg/paper-global.yml"} {
d, err := OpenDownload(root, name, false)
if err == nil {
d.Close()
t.Errorf("%s: opened for download", name)
} else if !strings.Contains(err.Error(), "forwarding secret") {
t.Errorf("%s: err = %v", name, err)
}
}
})
// A guarded name that is itself a link guards what it points at: that file
// is what the server reads, under whatever name it is reached.
t.Run("a guarded name that is a link guards its target", func(t *testing.T) {
root, _ := worldRoot(t)
for name, body := range map[string]string{
"config/real.yml": "secret: aVeryRealForwardingKey\n",
"real.properties": "motd=hi\nrcon.password=hunter2\n",
} {
if err := os.WriteFile(filepath.Join(root, name), []byte(body), 0o644); err != nil {
t.Fatal(err)
}
}
if err := os.Remove(filepath.Join(root, "server.properties")); err != nil {
t.Fatal(err)
}
symlink(t, "real.yml", filepath.Join(root, "config/paper-global.yml"))
symlink(t, "real.properties", filepath.Join(root, "server.properties"))
if d, err := OpenDownload(root, "config/real.yml", false); err == nil {
d.Close()
t.Error("the forwarding secret opened under its link target's name")
}
if _, got := download(t, root, "real.properties", false); string(got) != redacted {
t.Errorf("real.properties = %q, want %q", got, redacted)
}
})
t.Run("what is not there as the listing said is refused", func(t *testing.T) {
root := downloadWorld(t)
// Opening a FIFO for reading would wait for a writer that never comes.
if err := syscall.Mkfifo(filepath.Join(root, "pipe"), 0o644); err != nil {
t.Fatal(err)
}
for _, c := range []struct {
name string
dir bool
want string
}{
{"config", false, "is a folder now"},
{"server.properties", true, "is not a folder now"},
{"missing.txt", false, "no such file"},
{"pipe", false, "not a regular file"},
{".", true, "not a file or folder inside"},
{"", true, "not a file or folder inside"},
{"../outside", true, "not a file or folder inside"},
{"/etc", true, "not a file or folder inside"},
} {
d, err := OpenDownload(root, c.name, c.dir)
if err == nil {
d.Close()
t.Errorf("%q: opened", c.name)
} else if !strings.Contains(err.Error(), c.want) {
t.Errorf("%q: err = %v, want %q", c.name, err, c.want)
}
}
})
t.Run("a folder goes out as a zip under its own name, guarded", func(t *testing.T) {
root := downloadWorld(t)
plugins := filepath.Join(root, "plugins")
for _, d := range []string{"plugins/Essentials/empty", "plugins/Essentials/data"} {
if err := os.MkdirAll(filepath.Join(root, d), 0o755); err != nil {
t.Fatal(err)
}
}
for name, body := range map[string]string{
"plugins/a.jar": "jar",
"plugins/Essentials/config.yml": "x: 1",
"plugins/Essentials/data/server.properties": "rcon.password=notthereal\n",
} {
if err := os.WriteFile(filepath.Join(root, name), []byte(body), 0o644); err != nil {
t.Fatal(err)
}
}
for name, mode := range map[string]os.FileMode{"a.jar": 0o755, "Essentials/empty": 0o700} {
if err := os.Chmod(filepath.Join(plugins, name), mode); err != nil {
t.Fatal(err)
}
}
if err := os.Link(filepath.Join(root, "config/paper-global.yml"), filepath.Join(plugins, "stolen.yml")); err != nil {
t.Fatal(err)
}
if err := os.Link(filepath.Join(root, "server.properties"), filepath.Join(plugins, "props.txt")); err != nil {
t.Fatal(err)
}
symlink(t, "../config/paper-global.yml", filepath.Join(plugins, "sym.yml"))
d, b := download(t, root, "plugins", true)
if d.Size != -1 || d.ContentType != DownloadZipType {
t.Fatalf("size %d, type %s", d.Size, d.ContentType)
}
want := map[string]string{
"plugins/": "<dir 40755>",
"plugins/Essentials/": "<dir 40755>",
"plugins/Essentials/empty/": "<dir 40700>",
"plugins/Essentials/data/": "<dir 40755>",
"plugins/a.jar": "-rwxr-xr-x jar",
"plugins/Essentials/config.yml": "-rw-r--r-- x: 1",
// Only the world root's server.properties is the server's.
"plugins/Essentials/data/server.properties": "-rw-r--r-- rcon.password=notthereal\n",
"plugins/props.txt": "-rw-r--r-- " + redacted,
}
if got := unzipped(t, b); !reflect.DeepEqual(got, want) {
t.Fatalf("zip = %v\nwant %v", got, want)
}
if d.Skipped != 1 || d.Withheld != 1 {
t.Errorf("skipped %d, withheld %d; want 1 and 1", d.Skipped, d.Withheld)
}
if bytes.Contains(b, []byte("aVeryReal")) || bytes.Contains(b, []byte("hunter2")) {
t.Fatal("a secret is in the zip")
}
// A nested folder unpacks as itself, not under its parents.
_, b = download(t, root, "plugins/Essentials/data", true)
want = map[string]string{
"data/": "<dir 40755>",
"data/server.properties": "-rw-r--r-- rcon.password=notthereal\n",
}
if got := unzipped(t, b); !reflect.DeepEqual(got, want) {
t.Fatalf("nested zip = %v\nwant %v", got, want)
}
})
// Redaction reads the file whole; one too big for that is refused, or left
// out of a folder, never sent as it is.
t.Run("a server.properties too big to redact is refused", func(t *testing.T) {
root := downloadWorld(t)
big := append([]byte("rcon.password=hunter2\n"), bytes.Repeat([]byte("#"), MaxReadBytes)...)
if err := os.WriteFile(filepath.Join(root, "server.properties"), big, 0o644); err != nil {
t.Fatal(err)
}
d, err := OpenDownload(root, "server.properties", false)
if err == nil {
d.Close()
t.Fatal("an oversized server.properties opened for download")
}
if !strings.Contains(err.Error(), "cannot be redacted") {
t.Fatalf("err = %v", err)
}
if err := os.Link(filepath.Join(root, "server.properties"), filepath.Join(root, "config/props.txt")); err != nil {
t.Fatal(err)
}
d, b := download(t, root, "config", true)
want := map[string]string{"config/": "<dir 40755>", "config/paper.yml": "-rw-r--r-- verbose: false\n"}
if got := unzipped(t, b); !reflect.DeepEqual(got, want) || d.Withheld != 2 {
t.Fatalf("zip = %v, withheld %d; want %v and 2", got, d.Withheld, want)
}
})
t.Run("a file that shrinks mid-download fails it", func(t *testing.T) {
root := downloadWorld(t)
d, err := OpenDownload(root, "config/paper.yml", false)
if err != nil {
t.Fatal(err)
}
defer d.Close()
if err := os.Truncate(filepath.Join(root, "config/paper.yml"), 3); err != nil {
t.Fatal(err)
}
if err := d.WriteTo(context.Background(), io.Discard); err == nil || !strings.Contains(err.Error(), "shrank") {
t.Fatalf("err = %v, want the shrank error", err)
}
})
t.Run("a cancelled download stops", func(t *testing.T) {
root := downloadWorld(t)
// Folders only: no file copy is there to notice the cancel.
if err := os.MkdirAll(filepath.Join(root, "empty/a/b"), 0o755); err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithCancel(context.Background())
cancel()
for _, c := range []struct {
name string
dir bool
}{{"config/paper.yml", false}, {"config", true}, {"empty", true}} {
d, err := OpenDownload(root, c.name, c.dir)
if err != nil {
t.Fatal(err)
}
if err := d.WriteTo(ctx, io.Discard); err != context.Canceled {
t.Errorf("%s: err = %v, want context.Canceled", c.name, err)
}
d.Close()
}
})
}
+134 -25
View File
@@ -79,13 +79,12 @@ var (
ErrExists = errors.New("fileedit: the target already exists")
)
// Runner is the cluster-side half of one file operation: render and create the
// Job, wait for its Pod to reach a terminal phase, and return the marked JSON
// payload the Pod printed. It is one method rather than a create/poll/read trio
// because felis-api cannot poll a Job at all (no jobs:get — see FilesJobName), so
// there is no intermediate state a caller could usefully observe; the operation is
// synchronous from the API's point of view whether or not the seam pretends
// otherwise.
// Runner is the cluster-side half of a file operation. Run renders and creates
// the Job, waits for its Pod to reach a terminal phase, and returns the marked
// JSON payload the Pod printed: from the API's point of view an operation is one
// call. An upload too big for one request and an unzip can outlast any request,
// so those two are started instead (Start) and read back later (Ops), from the
// Jobs felis-api lists and the progress lines their Pods print.
//
// It is an interface so the Editor's orchestration and error mapping are tested
// against a fake; the client-go implementation (K8sRunner) is integration-only.
@@ -93,6 +92,11 @@ type Runner interface {
// Run creates the Job for p and returns the raw JSON payload from the
// ResultPrefix line of its Pod's log.
Run(ctx context.Context, p JobParams) ([]byte, error)
// Start creates the Job for p and returns once it exists.
Start(ctx context.Context, p JobParams) error
// Ops reports the background operations (JobParams.Async) of one server
// whose Jobs the cluster still holds, newest first.
Ops(ctx context.Context, namespace, server string) ([]OpState, error)
}
// Config parameterises the file editor. Image has no default on purpose: it is
@@ -139,6 +143,15 @@ type Config struct {
// it must stay comfortably longer than the moment felis-api needs to read the
// Pod's log, because the TTL takes the Pod (and its log) with the Job.
TTLAfterFinished time.Duration
// AsyncDeadline, AsyncTTL and AsyncCPULimit stand in for Deadline,
// TTLAfterFinished and CPULimit on an upload or unzip felis-api starts and
// does not wait on. Such a Job moves a whole archive or a file of gigabytes,
// so it gets hours; it stays after finishing long enough for the panel to
// show how it ended; and it gets a whole core, since inflating is CPU-bound.
AsyncDeadline time.Duration
AsyncTTL time.Duration
AsyncCPULimit string
}
// defaults applied when a Config field is left zero. They are sized for what a
@@ -153,6 +166,9 @@ const (
defaultCPULimit = "500m"
defaultMemLimit = "256Mi"
defaultTTL = 2 * time.Minute
defaultAsyncDeadline = 2 * time.Hour
defaultAsyncTTL = 30 * time.Minute
defaultAsyncCPULimit = "1"
)
// withDefaults returns a copy of c with zero fields filled, so a partially
@@ -182,6 +198,15 @@ func (c Config) withDefaults() Config {
if c.TTLAfterFinished <= 0 {
c.TTLAfterFinished = defaultTTL
}
if c.AsyncDeadline <= 0 {
c.AsyncDeadline = defaultAsyncDeadline
}
if c.AsyncTTL <= 0 {
c.AsyncTTL = defaultAsyncTTL
}
if c.AsyncCPULimit == "" {
c.AsyncCPULimit = defaultAsyncCPULimit
}
return c
}
@@ -191,19 +216,29 @@ type Editor struct {
Config Config
}
// Listing is one directory as List returns it.
type Listing struct {
Entries []Entry
// Truncated reports that the directory holds more than MaxEntries.
Truncated bool
// Free is the bytes free on the server's volume, 0 when the Job could not
// tell.
Free int64
}
// List returns one directory's entries, resolved under the server's world root.
// An empty path lists the world root itself.
func (e *Editor) List(ctx context.Context, server, path string) ([]Entry, bool, error) {
func (e *Editor) List(ctx context.Context, server, path string) (Listing, error) {
res, err := e.run(ctx, server, JobParams{Op: OpList, Path: path})
if err != nil {
return nil, false, err
return Listing{}, err
}
// A genuinely empty directory unmarshals Entries as nil; normalise it so the
// handler serialises [] rather than null.
if res.Entries == nil {
res.Entries = []Entry{}
}
return res.Entries, res.Truncated, nil
return Listing{Entries: res.Entries, Truncated: res.Truncated, Free: res.Avail}, nil
}
// Read returns a file's bytes, resolved under the server's world root, and the
@@ -289,7 +324,7 @@ func (e *Editor) run(ctx context.Context, server string, p JobParams) (Result, e
}
cfg := e.Config.withDefaults()
opID, err := newOpID()
p, err := cfg.params(server, p)
if err != nil {
return Result{}, err
}
@@ -301,20 +336,6 @@ func (e *Editor) run(ctx context.Context, server string, p JobParams) (Result, e
ctx, cancel := context.WithTimeout(ctx, cfg.Timeout)
defer cancel()
p.Server = server
p.OpID = opID
p.WorldPVC = naming.WorldPVCName(server)
p.Namespace = cfg.Namespace
p.ServiceAccount = cfg.ServiceAccount
p.Image = cfg.Image
p.WorldsRoot = cfg.WorldsRoot
p.Deadline = cfg.Deadline
p.CPULimit = cfg.CPULimit
p.MemLimit = cfg.MemLimit
p.RunAsUser = cfg.RunAsUser
p.RunAsGroup = cfg.RunAsGroup
p.FSGroup = cfg.FSGroup
p.TTLAfterFinished = cfg.TTLAfterFinished
payload, err := e.Runner.Run(ctx, p)
if err != nil {
return Result{}, err
@@ -327,6 +348,94 @@ func (e *Editor) run(ctx context.Context, server string, p JobParams) (Result, e
return res, resultError(res)
}
// params fills in what every Job of server takes from the Config, and a fresh op
// id; p carries the op and its own fields.
func (c Config) params(server string, p JobParams) (JobParams, error) {
opID, err := newOpID()
if err != nil {
return JobParams{}, err
}
p.Server = server
p.OpID = opID
p.WorldPVC = naming.WorldPVCName(server)
p.Namespace = c.Namespace
p.ServiceAccount = c.ServiceAccount
p.Image = c.Image
p.WorldsRoot = c.WorldsRoot
p.Deadline = c.Deadline
p.CPULimit = c.CPULimit
p.MemLimit = c.MemLimit
p.RunAsUser = c.RunAsUser
p.RunAsGroup = c.RunAsGroup
p.FSGroup = c.FSGroup
p.TTLAfterFinished = c.TTLAfterFinished
return p, nil
}
// The states of an OpState.
const (
OpRunning = "running"
OpSucceeded = "succeeded"
OpFailed = "failed"
)
// OpState is where one background file operation stands.
type OpState struct {
ID string
Op string
Path string
State string
// Started is when the Job was created; Finished when it ended, zero while it
// runs.
Started time.Time
Finished time.Time
// Done and Total are the bytes of the latest progress line, zero before the
// first.
Done, Total int64
// Result is what the Job printed once it finished. It is nil while the Job
// runs, and for a Job that ended without printing one (killed at its
// deadline, out of memory, its bytes unfetchable), whose Reason says why.
Result *Result
Reason string
}
// StartUpload starts landing the staged bytes src describes at path and returns
// without waiting, for a file too big to land inside one request (Upload). The
// Job checks what Upload's does; Ops reports how it ends.
func (e *Editor) StartUpload(ctx context.Context, server, path string, src UploadSource, overwrite bool) (OpState, error) {
return e.start(ctx, server, JobParams{
Op: OpUpload, Path: path, Overwrite: overwrite,
SourceURL: src.URL, UploadToken: src.Token, UploadSize: src.Size, UploadSHA256: src.SHA256,
})
}
// StartUnzip starts extracting the .zip at path into the folder holding it and
// returns without waiting. Without overwrite an archive that would replace a
// file changes nothing and ends with CodeExists and the list (Result.Conflicts).
func (e *Editor) StartUnzip(ctx context.Context, server, path string, overwrite bool) (OpState, error) {
return e.start(ctx, server, JobParams{Op: OpUnzip, Path: path, Overwrite: overwrite})
}
// Ops reports the server's background operations the cluster still holds: the
// one running, if any, and those finished within AsyncTTL.
func (e *Editor) Ops(ctx context.Context, server string) ([]OpState, error) {
return e.Runner.Ops(ctx, e.Config.withDefaults().Namespace, server)
}
func (e *Editor) start(ctx context.Context, server string, p JobParams) (OpState, error) {
cfg := e.Config.withDefaults()
p, err := cfg.params(server, p)
if err != nil {
return OpState{}, err
}
p.Async = true
p.Deadline, p.TTLAfterFinished, p.CPULimit = cfg.AsyncDeadline, cfg.AsyncTTL, cfg.AsyncCPULimit
if err := e.Runner.Start(ctx, p); err != nil {
return OpState{}, err
}
return OpState{ID: p.OpID, Op: p.Op, Path: p.Path, State: OpRunning, Started: time.Now()}, nil
}
// resultError translates a Result's code into the sentinel the API maps. An
// unrecognised code is deliberately NOT swallowed as success: a Job reporting a
// failure this build does not know about must still fail the request, or a future
+104 -6
View File
@@ -5,6 +5,7 @@ import (
"encoding/json"
"errors"
"testing"
"time"
)
// fakeRunner stands in for the cluster: it records the JobParams the Editor
@@ -14,6 +15,21 @@ type fakeRunner struct {
got []JobParams
payload []byte
err error
started []JobParams
startErr error
ops []OpState
opsArgs [][2]string
}
func (f *fakeRunner) Start(_ context.Context, p JobParams) error {
f.started = append(f.started, p)
return f.startErr
}
func (f *fakeRunner) Ops(_ context.Context, namespace, server string) ([]OpState, error) {
f.opsArgs = append(f.opsArgs, [2]string{namespace, server})
return f.ops, f.err
}
func (f *fakeRunner) Run(_ context.Context, p JobParams) ([]byte, error) {
@@ -37,15 +53,15 @@ func mustPayload(t *testing.T, res Result) []byte {
// the editor pointed at the same volume the operator created and the reaper deletes.
func TestEditorRendersParams(t *testing.T) {
t.Run("list", func(t *testing.T) {
r := &fakeRunner{payload: mustPayload(t, Result{Entries: []Entry{{Name: "a"}}})}
r := &fakeRunner{payload: mustPayload(t, Result{Entries: []Entry{{Name: "a"}}, Avail: 7 << 30})}
e := &Editor{Runner: r, Config: Config{Image: "img"}}
entries, truncated, err := e.List(context.Background(), "survival", "config")
ls, err := e.List(context.Background(), "survival", "config")
if err != nil {
t.Fatalf("List: %v", err)
}
if len(entries) != 1 || truncated {
t.Fatalf("entries=%+v truncated=%v", entries, truncated)
if len(ls.Entries) != 1 || ls.Truncated || ls.Free != 7<<30 {
t.Fatalf("listing = %+v", ls)
}
p := r.got[0]
if p.Op != OpList || p.Path != "config" || p.Server != "survival" {
@@ -242,11 +258,11 @@ func TestEditorNormalisesEmptyResults(t *testing.T) {
r := &fakeRunner{payload: mustPayload(t, Result{})}
e := &Editor{Runner: r, Config: Config{Image: "img"}}
entries, _, err := e.List(context.Background(), "survival", "empty")
ls, err := e.List(context.Background(), "survival", "empty")
if err != nil {
t.Fatalf("List: %v", err)
}
if entries == nil {
if ls.Entries == nil {
t.Fatal("an empty directory must list as [], not nil")
}
@@ -297,3 +313,85 @@ func TestExtractResult(t *testing.T) {
}
})
}
// TestEditorStartsBackgroundOps checks an upload or unzip too long to wait on is
// started, not run: its Job carries the async label and the longer deadline,
// the longer TTL Ops reads it back within, and the larger CPU share, and what
// comes back names the Job Ops will report on.
func TestEditorStartsBackgroundOps(t *testing.T) {
src := UploadSource{URL: "http://api/big", Token: "tok", Size: 5 << 30, SHA256: "sum"}
t.Run("upload", func(t *testing.T) {
r := &fakeRunner{}
e := &Editor{Runner: r, Config: Config{Image: "img", Namespace: "mc"}}
before := time.Now()
st, err := e.StartUpload(context.Background(), "survival", "maps/world.zip", src, true)
if err != nil {
t.Fatalf("StartUpload: %v", err)
}
if r.calls != 0 || len(r.started) != 1 {
t.Fatalf("ran %d, started %d; want the one Job started and none waited on", r.calls, len(r.started))
}
p := r.started[0]
if !p.Async || p.Op != OpUpload || p.Path != "maps/world.zip" || !p.Overwrite ||
p.SourceURL != src.URL || p.UploadToken != "tok" || p.UploadSize != 5<<30 || p.UploadSHA256 != "sum" {
t.Fatalf("params = %+v", p)
}
if p.Deadline != 2*time.Hour || p.TTLAfterFinished != 30*time.Minute || p.CPULimit != "1" || p.MemLimit != "256Mi" {
t.Fatalf("deadline %v ttl %v cpu %q mem %q, want 2h 30m 1 256Mi",
p.Deadline, p.TTLAfterFinished, p.CPULimit, p.MemLimit)
}
if p.Namespace != "mc" || p.Server != "survival" || p.WorldPVC != "world-survival-0" || p.Image != "img" || p.OpID == "" {
t.Fatalf("params = %+v", p)
}
if st.ID != p.OpID || st.Op != OpUpload || st.Path != "maps/world.zip" || st.State != OpRunning || st.Started.Before(before) {
t.Fatalf("state = %+v, want the started Job %s running", st, p.OpID)
}
if _, err := e.StartUpload(context.Background(), "survival", "maps/world.zip", src, false); err != nil || r.started[1].Overwrite {
t.Fatalf("an upload that must not replace a file started with %+v (%v)", r.started[1], err)
}
})
t.Run("unzip, with its own limits", func(t *testing.T) {
r := &fakeRunner{}
e := &Editor{Runner: r, Config: Config{Image: "img", AsyncDeadline: time.Hour, AsyncTTL: time.Minute, AsyncCPULimit: "2"}}
st, err := e.StartUnzip(context.Background(), "survival", "maps/world.zip", false)
if err != nil {
t.Fatalf("StartUnzip: %v", err)
}
p := r.started[0]
if !p.Async || p.Op != OpUnzip || p.Path != "maps/world.zip" || p.Overwrite || p.SourceURL != "" {
t.Fatalf("params = %+v", p)
}
if p.Deadline != time.Hour || p.TTLAfterFinished != time.Minute || p.CPULimit != "2" {
t.Fatalf("deadline %v ttl %v cpu %q, want the configured 1h 1m 2", p.Deadline, p.TTLAfterFinished, p.CPULimit)
}
if st.ID != p.OpID || st.Op != OpUnzip {
t.Fatalf("state = %+v", st)
}
})
t.Run("a Job that could not be created", func(t *testing.T) {
boom := errors.New("forbidden")
r := &fakeRunner{startErr: boom}
e := &Editor{Runner: r, Config: Config{Image: "img"}}
st, err := e.StartUnzip(context.Background(), "survival", "a.zip", false)
if !errors.Is(err, boom) || st != (OpState{}) {
t.Fatalf("state %+v err %v, want nothing started and %v", st, err, boom)
}
})
t.Run("ops", func(t *testing.T) {
want := []OpState{{ID: "0a", State: OpRunning}}
r := &fakeRunner{ops: want}
e := &Editor{Runner: r, Config: Config{Image: "img"}}
got, err := e.Ops(context.Background(), "survival")
if err != nil || len(got) != 1 || got[0] != want[0] {
t.Fatalf("Ops = %+v %v", got, err)
}
if r.opsArgs[0] != [2]string{"minecraft", "survival"} {
t.Fatalf("asked %v, want the default namespace and the server", r.opsArgs[0])
}
})
}
+81 -29
View File
@@ -10,6 +10,7 @@ import (
"fmt"
"io"
"io/fs"
"math"
"os"
"path"
"strings"
@@ -20,11 +21,12 @@ import (
)
// The operations the editor supports: list a directory, read a file, write a
// file, make a directory, delete, rename, and upload. The set is closed; there is
// no chmod, chown, link or copy. Every op resolves every path through os.Root (see
// Execute), and each mutating op carries its own containment note below.
// file, make a directory, delete, rename, upload, and extract a .zip. The set is
// closed; there is no chmod, chown, link or copy. Every op resolves every path
// through os.Root (see Execute), and each mutating op carries its own
// containment note below.
//
// A write or upload DOES land arbitrary bytes at any path inside the mount, and
// A write, upload or unzip DOES land arbitrary bytes at any path inside the mount, and
// that is a real capability rather than an oversight: the root is the server's
// whole working directory (see Config.WorldsRoot), so an owner can upload
// plugins/<x>.jar and Paper will load it on the next boot. It is the same power a
@@ -33,8 +35,8 @@ import (
// image curation. Images are admin-only (POST /images, POST /images/build) and
// modpack submissions need an admin verdict, so this is the one owner-tier route
// that lands executable code in a backend pod. That trade was made deliberately;
// if it is ever revisited, the guard belongs in land() below, which is the single
// choke point both byte-landing ops route through.
// if it is ever revisited, the guard belongs in land() below, the choke point
// write and upload route through, and in unzip's extractOne (unzip.go).
const (
OpList = "list"
OpRead = "read"
@@ -43,6 +45,7 @@ const (
OpDelete = "delete"
OpRename = "rename"
OpUpload = "upload"
OpUnzip = "unzip"
)
// mutates reports whether op changes the world, and so whether its Job gets the
@@ -54,7 +57,7 @@ func mutates(op string) bool { return op != OpList && op != OpRead }
// validOp reports whether op is one the Job knows.
func validOp(op string) bool {
switch op {
case OpList, OpRead, OpWrite, OpMkdir, OpDelete, OpRename, OpUpload:
case OpList, OpRead, OpWrite, OpMkdir, OpDelete, OpRename, OpUpload, OpUnzip:
return true
}
return false
@@ -133,12 +136,12 @@ const UploadTokenEnv = "FELIS_UPLOAD_TOKEN"
// - MaxEntries bounds a listing. A world's region/ directory legitimately holds
// thousands of .mca files, so this truncates rather than errors (Truncated
// says so), keeping the log line bounded while still being useful.
// - MaxUploadBytes bounds an upload. Its bytes travel neither through the Job
// spec nor the pod log — felis-api stages them and the Job fetches them — so
// the bound is the request body instead: the Cloudflare edge refuses bodies
// over 100 MB on the Free and Pro plans, and 64 MiB covers the largest plugin
// jars (a Geyser build is about 20 MiB) with room to spare. A whole world is
// a different operation (a restore), not an upload.
// - MaxUploadBytes bounds an upload sent as ONE request body: the Cloudflare
// edge refuses bodies over 100 MB on the Free and Pro plans, and 64 MiB
// covers the largest plugin jars (a Geyser build is about 20 MiB) with room
// to spare. It is felis-api's bound on that route only. A bigger file arrives
// in parts and is bounded by nothing but the room on the server's volume,
// which the Job checks before it fetches a byte (upload).
const (
MaxWriteBytes = 256 << 10 // 256 KiB
MaxReadBytes = 1 << 20 // 1 MiB
@@ -183,6 +186,22 @@ type Result struct {
// conflict, the file as it is now. A client hands it back as the expected
// hash of its next write (see write).
SHA256 string `json:"sha256,omitempty"`
// Conflicts lists, relative to the root and sorted, the existing files an
// unzip would replace: the first MaxConflicts of them. ConflictCount is how
// many there are in all.
Conflicts []string `json:"conflicts,omitempty"`
ConflictCount int `json:"conflict_count,omitempty"`
// Entry names what an unzip refused: the archive entry, or the path on the
// server it collides with.
Entry string `json:"entry,omitempty"`
// Need and Avail are, on a no_space an upload or unzip saw coming, the bytes
// it needs and the bytes the volume has free. A listing sets Avail too.
Need int64 `json:"need,omitempty"`
Avail int64 `json:"avail,omitempty"`
// Files and Bytes are what a successful unzip extracted.
Files int `json:"files,omitempty"`
Bytes int64 `json:"bytes,omitempty"`
}
// Request is one file operation. Op decides which of the other fields it reads.
@@ -200,10 +219,15 @@ type Request struct {
// panel's "new file", which must never truncate a file it did not know was
// there.
CreateOnly bool
// Upload is where an upload's bytes come from; Overwrite lets it replace a
// file already at the path.
// Upload is where an upload's bytes come from. Overwrite lets an upload
// replace a file already at the path, and an unzip replace the files it
// collides with.
Upload *Upload
Overwrite bool
// Progress, when set, hears how far an upload or unzip has got: bytes landed
// so far out of the total. It is called from the copy loop, often; the caller
// throttles.
Progress func(done, total int64)
}
// Upload describes the bytes an upload lands. Size and SHA256 are what felis-api
@@ -262,7 +286,7 @@ func Execute(root string, req Request) (Result, error) {
switch req.Op {
case OpList:
return list(r, path), nil
return list(r, root, path), nil
case OpRead:
return read(r, path), nil
case OpWrite:
@@ -277,7 +301,9 @@ func Execute(root string, req Request) (Result, error) {
if req.Upload == nil {
return Result{}, errors.New("an upload needs a source")
}
return upload(r, path, *req.Upload, req.Overwrite)
return upload(r, root, path, *req.Upload, req.Overwrite, req.Progress)
case OpUnzip:
return unzip(r, root, path, req.Overwrite, req.Progress), nil
default:
return Result{}, fmt.Errorf("unknown op %q", req.Op)
}
@@ -285,8 +311,10 @@ func Execute(root string, req Request) (Result, error) {
// list reads one directory. It does not recurse: a browser asks for one level at
// a time, and recursion would make both the result size and the traversal cost
// unbounded in a world directory.
func list(r *os.Root, path string) Result {
// unbounded in a world directory. It also reports the room left on the volume
// (Avail), so the panel can refuse an upload the volume cannot take before
// sending a byte of it.
func list(r *os.Root, rootPath, path string) Result {
f, err := r.Open(path)
if err != nil {
return failure(err, path)
@@ -318,7 +346,11 @@ func list(r *os.Root, path string) Result {
}
entries = append(entries, e)
}
return Result{Entries: entries, Truncated: truncated}
res := Result{Entries: entries, Truncated: truncated}
if avail, _, err := statfs(rootPath); err == nil {
res.Avail = int64(min(avail, math.MaxInt64))
}
return res
}
// secretConfigPath is the one file in a world mount holding PLATFORM secret
@@ -370,6 +402,13 @@ func read(r *os.Root, name string) Result {
if info.IsDir() {
return Result{Code: CodeBadPath, Error: fmt.Sprintf("%s is a directory, not a file", name)}
}
// The name check above answers the plain path with a clear reason; this one
// catches the same file reached through a link (see Guard).
withhold, redact := NewGuard(r).Rule(info)
if withhold {
return Result{Code: CodeBadPath, Error: fmt.Sprintf(
"%s is the file holding the proxy forwarding secret, which is shared cluster-wide, and is not readable through the editor", name)}
}
if info.Size() > MaxReadBytes {
return Result{Code: CodeTooLarge, Error: fmt.Sprintf(
"%s is %d bytes; the editor reads at most %d", name, info.Size(), MaxReadBytes)}
@@ -382,7 +421,11 @@ func read(r *os.Root, name string) Result {
if err != nil {
return failure(err, name)
}
return Result{Content: redactSecretProps(name, b), SHA256: digest(b)}
content := b
if redact {
content = RedactProps(b)
}
return Result{Content: content, SHA256: digest(b)}
}
// propsPath is the server's main config file, and rconPasswordKey the one line in
@@ -515,8 +558,8 @@ func (e *transferError) Error() string { return "fetch upload: " + e.err.Error()
func (e *transferError) Unwrap() error { return e.err }
// land atomically puts the bytes fill writes at target, the path landingTarget
// returned for name. It is the single choke point both byte-landing ops (write and
// upload) route through.
// returned for name. Write and upload both land through it; unzip lands a whole
// tree at once and has its own path (unzip.go).
//
// The bytes go to a temporary sibling that is synced and then renamed over the
// target, so a full disk, a Job killed at its deadline or a crashed node leaves
@@ -596,15 +639,19 @@ func syncDir(r *os.Root, dir string) {
// The fetched bytes must match both the size and the SHA-256 felis-api received;
// either mismatch is a broken transfer, and the target is left as it was. A
// success has landed exactly what felis-api staged, whose digest it already holds.
func upload(r *os.Root, name string, u Upload, overwrite bool) (Result, error) {
if u.Size > MaxUploadBytes {
return Result{Code: CodeTooLarge, Error: fmt.Sprintf(
"the upload is %d bytes; the editor uploads at most %d", u.Size, MaxUploadBytes)}, nil
}
//
// There is no size ceiling here. The volume is the bound, and it is checked up
// front: the new bytes land beside the file they replace until the rename, so
// they need their whole size free whatever is already at the path.
func upload(r *os.Root, rootPath, name string, u Upload, overwrite bool, progress func(done, total int64)) (Result, error) {
target, mode, res := landingTarget(r, name, overwrite)
if res.Code != "" {
return res, nil
}
if avail, _, err := statfs(rootPath); err == nil && uint64(u.Size) > avail {
return Result{Code: CodeNoSpace, Need: u.Size, Avail: int64(min(avail, math.MaxInt64)), Error: fmt.Sprintf(
"%s is %d bytes and the server's volume has %d free; nothing was changed", name, u.Size, avail)}, nil
}
return land(r, name, target, mode, func(w io.Writer) error {
body, err := u.Open()
if err != nil {
@@ -612,8 +659,13 @@ func upload(r *os.Root, name string, u Upload, overwrite bool) (Result, error) {
}
defer body.Close()
h := sha256.New()
var done int64
out := io.MultiWriter(w, h)
if progress != nil {
out = countingWriter{out, func(n int) { done += int64(n); progress(done, u.Size) }}
}
// One byte past Size so a source that sends more than it promised is seen.
n, err := io.Copy(io.MultiWriter(w, h), sourceReader{io.LimitReader(body, u.Size+1)})
n, err := io.Copy(out, sourceReader{io.LimitReader(body, u.Size+1)})
if err != nil {
return err
}
+79
View File
@@ -2,7 +2,11 @@ package fileedit
import (
"bytes"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"math"
"os"
"path/filepath"
"strings"
@@ -176,6 +180,31 @@ func TestExecuteHappyPath(t *testing.T) {
}
})
t.Run("a listing reports the room left on the volume", func(t *testing.T) {
prev := statfs
t.Cleanup(func() { statfs = prev })
var asked string
statfs = func(dir string) (uint64, uint64, error) { asked = dir; return 12345, 99999, nil }
res, err := run(root, OpList, "config", nil, "")
if err != nil || res.Code != "" {
t.Fatalf("Execute: %v %+v", err, res)
}
if res.Avail != 12345 || asked != root {
t.Fatalf("avail = %d measured at %q, want 12345 at %q", res.Avail, asked, root)
}
statfs = func(string) (uint64, uint64, error) { return math.MaxUint64, math.MaxUint64, nil }
if res, _ := run(root, OpList, "config", nil, ""); res.Avail != math.MaxInt64 {
t.Fatalf("avail = %d, want it clamped to %d", res.Avail, int64(math.MaxInt64))
}
statfs = func(string) (uint64, uint64, error) { return 1, 1, errors.New("no statfs") }
res, err = run(root, OpList, "config", nil, "")
if err != nil || res.Code != "" || len(res.Entries) != 1 || res.Avail != 0 {
t.Fatalf("a volume that cannot be measured still lists, with no room reported: %v %+v", err, res)
}
})
t.Run("read a file", func(t *testing.T) {
res, err := run(root, OpRead, "server.properties", nil, "")
if err != nil {
@@ -396,6 +425,11 @@ func TestReadRedactsRconPassword(t *testing.T) {
if err != nil {
t.Fatalf("Execute: %v", err)
}
// The hash is the file on disk, the one a save's expect_sha256 is checked
// against, not the redacted copy the editor shows.
if sum := sha256.Sum256([]byte(props)); res.SHA256 != hex.EncodeToString(sum[:]) {
t.Fatalf("sha256 = %s, want the hash of the file as stored", res.SHA256)
}
got := string(res.Content)
if strings.Contains(got, "hunter2") {
t.Fatalf("read returned the RCON password (spec §286):\n%s", got)
@@ -420,6 +454,51 @@ func TestReadRedactsRconPassword(t *testing.T) {
}
}
// TestReadGuardsLinksToGuardedFiles: a plugin runs as the game uid and can leave
// a link to either guarded file anywhere in the world. Read under the link's
// name, the forwarding secret is still refused and the RCON password still
// redacted, whether the link is symbolic, a hard link, or a linked folder.
func TestReadGuardsLinksToGuardedFiles(t *testing.T) {
root, _ := worldRoot(t)
const secret = "secret: aVeryRealForwardingKey"
if err := os.WriteFile(filepath.Join(root, "config", "paper-global.yml"), []byte(secret), 0o644); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(root, "server.properties"), []byte("motd=hi\nrcon.password=hunter2\n"), 0o644); err != nil {
t.Fatal(err)
}
symlink(t, "config/paper-global.yml", filepath.Join(root, "sym.yml"))
symlink(t, "config", filepath.Join(root, "cfg-link"))
symlink(t, "server.properties", filepath.Join(root, "sym.properties"))
for _, l := range [][2]string{
{"config/paper-global.yml", "hard.yml"},
{"server.properties", "hard.properties"},
} {
if err := os.Link(filepath.Join(root, l[0]), filepath.Join(root, l[1])); err != nil {
t.Fatal(err)
}
}
for _, name := range []string{"sym.yml", "cfg-link/paper-global.yml", "hard.yml"} {
res, err := run(root, OpRead, name, nil, "")
if err != nil {
t.Fatalf("%s: %v", name, err)
}
if res.Code != CodeBadPath || strings.Contains(string(res.Content), "aVeryReal") {
t.Errorf("%s: result = %+v, want bad_path and no secret", name, res)
}
}
for _, name := range []string{"sym.properties", "hard.properties"} {
res, err := run(root, OpRead, name, nil, "")
if err != nil {
t.Fatalf("%s: %v", name, err)
}
if want := "motd=hi\nrcon.password=" + redactedValue + "\n"; res.Code != "" || string(res.Content) != want {
t.Errorf("%s: result = %+v, want content %q", name, res, want)
}
}
}
// TestWriteIsAtomic is the durability contract: a write that fails part-way leaves
// the original file byte-for-byte intact and no stray sibling behind, and a write
// that succeeds keeps the file's mode.
+58
View File
@@ -0,0 +1,58 @@
package fileedit
import (
"io/fs"
"os"
"path"
)
// What leaves a world mount — a read, a download, a world export, a backup
// export — passes the same two guards: the forwarding-secret file
// (secretConfigPath) is withheld, and server.properties has its RCON password
// redacted (propsPath).
//
// On a live mount both are matched by the file itself (os.SameFile), not by the
// name it was reached under. A plugin runs arbitrary code as the game uid and can
// leave a symbolic or hard link to either file anywhere in the world; a name
// check alone would hand the secret out under the link's name. A stored archive
// has only names, so ArchiveRule matches those.
// Guard knows the two guarded files of one world mount.
type Guard struct {
secret, props fs.FileInfo
}
// NewGuard looks the guarded files up in r. One that is missing guards nothing:
// no file can be the same file as it.
func NewGuard(r *os.Root) Guard {
var g Guard
if fi, err := r.Stat(secretConfigPath); err == nil {
g.secret = fi
}
if fi, err := r.Stat(propsPath); err == nil {
g.props = fi
}
return g
}
// Rule reports whether the file fi describes must be withheld, or sent only
// through RedactProps.
func (g Guard) Rule(fi fs.FileInfo) (withhold, redact bool) {
if g.secret != nil && os.SameFile(g.secret, fi) {
return true, false
}
return false, g.props != nil && os.SameFile(g.props, fi)
}
// ArchiveRule is Rule for an entry of a stored world archive, by its name
// cleaned as a path, so "./server.properties" is server.properties too.
func ArchiveRule(name string) (withhold, redact bool) {
name = path.Clean(name)
return name == secretConfigPath, name == propsPath
}
// RedactProps replaces the RCON password in server.properties content with
// redactedValue (see redactSecretProps for why a placeholder and not a blank).
func RedactProps(content []byte) []byte {
return redactSecretProps(propsPath, content)
}
+25
View File
@@ -0,0 +1,25 @@
package fileedit
import "testing"
// TestArchiveRule: a stored archive's entry is matched by its name as a path,
// however the archive spelled it.
func TestArchiveRule(t *testing.T) {
for _, c := range []struct {
name string
withhold, redact bool
}{
{"config/paper-global.yml", true, false},
{"./config/paper-global.yml", true, false},
{"config//paper-global.yml", true, false},
{"server.properties", false, true},
{"./server.properties", false, true},
{"plugins/server.properties", false, false},
{"plugins/config/paper-global.yml", false, false},
{"config/paper.yml", false, false},
} {
if w, r := ArchiveRule(c.name); w != c.withhold || r != c.redact {
t.Errorf("ArchiveRule(%q) = %v, %v; want %v, %v", c.name, w, r, c.withhold, c.redact)
}
}
}
+40 -10
View File
@@ -25,6 +25,11 @@ const (
LabelServer = "felis.lolicon.best/server"
LabelOpID = "felis.lolicon.best/files-op"
LabelMode = "felis.lolicon.best/files-mode"
// LabelAsync marks the Job of an upload or unzip felis-api started and does
// not wait on (Editor.StartUpload, Editor.StartUnzip); Ops finds them by it.
// AnnotationPath names the file such a Job works on, for Ops to show.
LabelAsync = "felis.lolicon.best/files-async"
AnnotationPath = "felis.lolicon.best/files-path"
managedByValue = "felis-files"
componentValue = "world-files"
@@ -59,7 +64,11 @@ type JobParams struct {
UploadSize int64
UploadSHA256 string
Overwrite bool
WorldPVC string
// Async marks a Job felis-api does not wait on: it carries LabelAsync and
// AnnotationPath, which Ops reads it back by. Only an upload or an unzip
// runs so.
Async bool
WorldPVC string
Namespace string
ServiceAccount string
@@ -92,13 +101,17 @@ type JobParams struct {
func FilesJobName(server, opID string) string { return "files-" + server + "-" + opID }
func filesLabels(p JobParams) map[string]string {
return map[string]string{
l := map[string]string{
LabelManagedBy: managedByValue,
LabelComponent: componentValue,
LabelServer: p.Server,
LabelOpID: p.OpID,
LabelMode: p.Op,
}
if p.Async {
l[LabelAsync] = "true"
}
return l
}
// FilesJob renders the file-editor Job. Its isolation is the strictest of the three
@@ -152,6 +165,9 @@ func FilesJob(p JobParams) (*batchv1.Job, error) {
if p.Op == OpUpload && (p.SourceURL == "" || p.UploadToken == "") {
return nil, fmt.Errorf("fileedit: an upload needs a source URL and a token")
}
if p.Async && p.Op != OpUpload && p.Op != OpUnzip {
return nil, fmt.Errorf("fileedit: only an upload or an unzip runs in the background, not %s", p.Op)
}
limits, err := resourceLimits(p.CPULimit, p.MemLimit)
if err != nil {
return nil, err
@@ -195,6 +211,10 @@ func FilesJob(p JobParams) (*batchv1.Job, error) {
if p.Overwrite {
args = append(args, "--overwrite")
}
case OpUnzip:
if p.Overwrite {
args = append(args, "--overwrite")
}
}
container := corev1.Container{
Name: containerName,
@@ -235,9 +255,10 @@ func FilesJob(p JobParams) (*batchv1.Job, error) {
job := &batchv1.Job{
ObjectMeta: metav1.ObjectMeta{
Name: FilesJobName(p.Server, p.OpID),
Namespace: p.Namespace,
Labels: filesLabels(p),
Name: FilesJobName(p.Server, p.OpID),
Namespace: p.Namespace,
Labels: filesLabels(p),
Annotations: filesAnnotations(p),
},
Spec: batchv1.JobSpec{
// One shot: a file operation that failed must surface its failure, not be
@@ -297,12 +318,12 @@ func int64Ptr(i int64) *int64 { return &i }
// filesCapabilities is what the root executor keeps after dropping ALL (see
// Config.RunAsUser). DAC_OVERRIDE opens a mode-0600 file (level.dat) the game wrote
// as its own uid, which a fixed non-root uid could not. A write, mkdir or upload
// also keeps CHOWN so what it creates can be handed to naming.GameUID (exec.go
// ownWritten). List, read, delete and rename create nothing and get no more than
// they need.
// as its own uid, which a fixed non-root uid could not. A write, mkdir, upload or
// unzip also keeps CHOWN so what it creates can be handed to naming.GameUID
// (exec.go ownWritten). List, read, delete and rename create nothing and get no
// more than they need.
func filesCapabilities(op string) []corev1.Capability {
if op == OpWrite || op == OpMkdir || op == OpUpload {
if op == OpWrite || op == OpMkdir || op == OpUpload || op == OpUnzip {
return []corev1.Capability{"CHOWN", "DAC_OVERRIDE"}
}
return []corev1.Capability{"DAC_OVERRIDE"}
@@ -323,3 +344,12 @@ func filesPodSecurityContext(p JobParams) *corev1.PodSecurityContext {
}
return sc
}
// filesAnnotations names the file an async Job works on. A path does not fit a
// label (63 characters, no slashes), so it rides an annotation.
func filesAnnotations(p JobParams) map[string]string {
if !p.Async {
return nil
}
return map[string]string{AnnotationPath: p.Path}
}
+38 -1
View File
@@ -128,7 +128,7 @@ func TestFilesJobIsolation(t *testing.T) {
chown bool
}{
{OpList, false}, {OpRead, false}, {OpDelete, false}, {OpRename, false},
{OpWrite, true}, {OpMkdir, true}, {OpUpload, true},
{OpWrite, true}, {OpMkdir, true}, {OpUpload, true}, {OpUnzip, true},
} {
j, err := FilesJob(opParams(tc.op))
if err != nil {
@@ -216,6 +216,7 @@ func TestFilesJobWorldMountIsReadOnlyForReads(t *testing.T) {
{OpDelete, false},
{OpRename, false},
{OpUpload, false},
{OpUnzip, false},
}
for _, tc := range cases {
t.Run(tc.op, func(t *testing.T) {
@@ -435,6 +436,9 @@ func TestFilesJobRejectsBadParams(t *testing.T) {
p.Op, p.Content = OpWrite, make([]byte, MaxWriteBytes+1)
}},
{"bad cpu limit", func(p *JobParams) { p.CPULimit = "half" }},
{"a read in the background", func(p *JobParams) { p.Async = true }},
{"a write in the background", func(p *JobParams) { p.Op, p.Async = OpWrite, true }},
{"a delete in the background", func(p *JobParams) { p.Op, p.Async = OpDelete, true }},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
@@ -446,3 +450,36 @@ func TestFilesJobRejectsBadParams(t *testing.T) {
})
}
}
// TestFilesJobAsync checks a background Job is marked so Ops finds it, on the
// Job and on its Pod, and carries the path Ops shows; a Job felis-api waits on
// carries neither, so Ops never reports it.
func TestFilesJobAsync(t *testing.T) {
for _, op := range []string{OpUpload, OpUnzip} {
t.Run(op, func(t *testing.T) {
p := opParams(op)
p.Path, p.Async = "maps/world.zip", true
j, err := FilesJob(p)
if err != nil {
t.Fatalf("FilesJob: %v", err)
}
if j.Labels[LabelAsync] != "true" || j.Spec.Template.Labels[LabelAsync] != "true" {
t.Fatalf("job labels %v, pod labels %v, want %s=true on both", j.Labels, j.Spec.Template.Labels, LabelAsync)
}
if len(j.Annotations) != 1 || j.Annotations[AnnotationPath] != "maps/world.zip" {
t.Fatalf("annotations = %v, want only %s", j.Annotations, AnnotationPath)
}
p.Async = false
j, err = FilesJob(p)
if err != nil {
t.Fatalf("FilesJob: %v", err)
}
_, onJob := j.Labels[LabelAsync]
_, onPod := j.Spec.Template.Labels[LabelAsync]
if onJob || onPod || j.Annotations != nil {
t.Fatalf("a Job waited on is labelled %v / %v and annotated %v", j.Labels, j.Spec.Template.Labels, j.Annotations)
}
})
}
}
+133
View File
@@ -3,12 +3,15 @@ package fileedit
import (
"bufio"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"sort"
"strings"
"time"
batchv1 "k8s.io/api/batch/v1"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes"
@@ -185,3 +188,133 @@ func tail(log string) string {
}
return "..." + log[len(log)-n:]
}
// Start creates the Job for p and returns without waiting on it. Ops reads it
// back.
func (k *K8sRunner) Start(ctx context.Context, p JobParams) error {
job, err := FilesJob(p)
if err != nil {
return err
}
if _, err := k.cs.BatchV1().Jobs(p.Namespace).Create(ctx, job, metav1.CreateOptions{}); err != nil {
return fmt.Errorf("fileedit: create file job: %w", err)
}
return nil
}
// maxOps bounds what Ops reports, and so how many Pod logs one call reads. Only
// one background op runs per server at a time (it holds the world volume), so
// this many are the one running and the latest that finished within AsyncTTL.
const maxOps = 10
// opLogLines is how much of an op's log Ops reads: the result line is the last
// thing the Job prints to stdout, the progress lines come before it, and a
// failed run ends with one line on stderr.
const opLogLines = 20
// Ops lists the server's background Jobs, newest first, with how far each has
// got and how it ended, read from the tail of its Pod's log. A Pod whose log
// cannot be read yet (still pulling its image) or any more (its node went away)
// reports no progress rather than failing the listing.
func (k *K8sRunner) Ops(ctx context.Context, namespace, server string) ([]OpState, error) {
sel := metav1.ListOptions{LabelSelector: LabelManagedBy + "=" + managedByValue + "," +
LabelServer + "=" + server + "," + LabelAsync + "=true"}
jobs, err := k.cs.BatchV1().Jobs(namespace).List(ctx, sel)
if err != nil {
return nil, fmt.Errorf("fileedit: list file jobs: %w", err)
}
pods, err := k.cs.CoreV1().Pods(namespace).List(ctx, sel)
if err != nil {
return nil, fmt.Errorf("fileedit: list file job pods: %w", err)
}
// backoffLimit is 0, so a Job has one Pod; the newest wins all the same.
podOf := map[string]*corev1.Pod{}
for i := range pods.Items {
pod := &pods.Items[i]
id := pod.Labels[LabelOpID]
if cur := podOf[id]; cur == nil || pod.CreationTimestamp.After(cur.CreationTimestamp.Time) {
podOf[id] = pod
}
}
items := jobs.Items
sort.SliceStable(items, func(i, j int) bool {
return items[i].CreationTimestamp.After(items[j].CreationTimestamp.Time)
})
if len(items) > maxOps {
items = items[:maxOps]
}
out := make([]OpState, 0, len(items))
for i := range items {
log := ""
if pod := podOf[items[i].Labels[LabelOpID]]; pod != nil && pod.Status.Phase != corev1.PodPending {
log, _ = k.logTail(ctx, namespace, pod.Name)
}
out = append(out, opState(&items[i], log))
}
return out, nil
}
// logTail reads the last opLogLines lines of a Pod's log.
func (k *K8sRunner) logTail(ctx context.Context, namespace, pod string) (string, error) {
lines := int64(opLogLines)
stream, err := k.cs.CoreV1().Pods(namespace).GetLogs(pod, &corev1.PodLogOptions{
Container: containerName, TailLines: &lines,
}).Stream(ctx)
if err != nil {
return "", err
}
defer stream.Close()
b, err := io.ReadAll(io.LimitReader(stream, maxLogBytes))
return string(b), err
}
// opState reads one background Job, and the tail of its Pod's log, as an
// OpState. The printed result decides the outcome whatever the Job's condition
// says: a Job killed at its deadline just after printing did finish its work.
// A Job that ended without one failed, and the condition's reason says how
// (DeadlineExceeded, BackoffLimitExceeded); a Job that completed but whose log
// could not be read has an outcome no one can tell, ResultUnavailable.
func opState(job *batchv1.Job, log string) OpState {
st := OpState{
ID: job.Labels[LabelOpID], Op: job.Labels[LabelMode], Path: job.Annotations[AnnotationPath],
State: OpRunning, Started: job.CreationTimestamp.Time,
}
if p, ok := lastProgress(log); ok {
st.Done, st.Total = p.Done, p.Total
}
ended, reason := false, ""
for _, c := range job.Status.Conditions {
if c.Status != corev1.ConditionTrue {
continue
}
switch c.Type {
case batchv1.JobComplete, batchv1.JobSuccessCriteriaMet:
ended, reason = true, "ResultUnavailable"
st.Finished = c.LastTransitionTime.Time
case batchv1.JobFailed, batchv1.JobFailureTarget:
ended, reason = true, c.Reason
st.Finished = c.LastTransitionTime.Time
}
}
if !ended {
return st
}
if payload, ok := extractResult(log); ok {
var res Result
if json.Unmarshal(payload, &res) == nil {
st.Result = &res
}
}
switch {
case st.Result != nil && st.Result.Code == "":
st.State = OpSucceeded
case st.Result != nil:
st.State = OpFailed
default:
st.State, st.Reason = OpFailed, reason
if st.Reason == "" {
st.Reason = "Failed"
}
}
return st
}
+241
View File
@@ -0,0 +1,241 @@
package fileedit
import (
"context"
"fmt"
"testing"
"time"
batchv1 "k8s.io/api/batch/v1"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/client-go/kubernetes/fake"
k8stesting "k8s.io/client-go/testing"
)
var (
opCreated = time.Date(2026, 9, 28, 10, 0, 0, 0, time.UTC)
opEnded = opCreated.Add(3 * time.Minute)
)
func asyncJob(conds ...batchv1.JobCondition) *batchv1.Job {
return &batchv1.Job{
ObjectMeta: metav1.ObjectMeta{
Name: "files-survival-0a",
CreationTimestamp: metav1.NewTime(opCreated),
Labels: map[string]string{LabelOpID: "0a", LabelMode: OpUnzip},
Annotations: map[string]string{AnnotationPath: "maps/world.zip"},
},
Status: batchv1.JobStatus{Conditions: conds},
}
}
func cond(typ batchv1.JobConditionType, status corev1.ConditionStatus, reason string) batchv1.JobCondition {
return batchv1.JobCondition{Type: typ, Status: status, Reason: reason, LastTransitionTime: metav1.NewTime(opEnded)}
}
// TestOpState pins how a background Job and the tail of its log read as an
// OpState: the printed result decides the outcome whatever the Job's condition
// says, and a Job that ended without one failed for the condition's reason.
func TestOpState(t *testing.T) {
progress := ProgressPrefix + `{"done":10,"total":100}` + "\n" +
"a stderr line\n" +
ProgressPrefix + `{"done":40,"total":100}` + "\n"
ok := ResultPrefix + `{"files":3,"bytes":40}` + "\n"
conflict := ResultPrefix + `{"code":"exists","conflicts":["a.txt"],"conflict_count":1}` + "\n"
complete := cond(batchv1.JobComplete, corev1.ConditionTrue, "")
cases := []struct {
name string
conds []batchv1.JobCondition
log string
state string
reason string
code string
files int
done int64
finished bool
wantResult bool
}{
{name: "running, at its latest progress", log: progress, state: OpRunning, done: 40},
{name: "running, before any progress", state: OpRunning},
{name: "a condition not yet true is still running",
conds: []batchv1.JobCondition{cond(batchv1.JobFailed, corev1.ConditionFalse, "DeadlineExceeded")},
log: progress, state: OpRunning, done: 40},
{name: "complete with a clean result",
conds: []batchv1.JobCondition{complete}, log: progress + ok,
state: OpSucceeded, files: 3, done: 40, finished: true, wantResult: true},
{name: "success criteria met before complete",
conds: []batchv1.JobCondition{cond(batchv1.JobSuccessCriteriaMet, corev1.ConditionTrue, "")}, log: ok,
state: OpSucceeded, files: 3, finished: true, wantResult: true},
{name: "killed at its deadline after printing a clean result",
conds: []batchv1.JobCondition{cond(batchv1.JobFailed, corev1.ConditionTrue, "DeadlineExceeded")}, log: ok,
state: OpSucceeded, files: 3, finished: true, wantResult: true},
{name: "complete with a refusal",
conds: []batchv1.JobCondition{complete}, log: conflict,
state: OpFailed, code: CodeExists, finished: true, wantResult: true},
{name: "killed at its deadline without a result",
conds: []batchv1.JobCondition{cond(batchv1.JobFailed, corev1.ConditionTrue, "DeadlineExceeded")}, log: progress,
state: OpFailed, reason: "DeadlineExceeded", done: 40, finished: true},
{name: "failure target before failed",
conds: []batchv1.JobCondition{cond(batchv1.JobFailureTarget, corev1.ConditionTrue, "BackoffLimitExceeded")},
state: OpFailed, reason: "BackoffLimitExceeded", finished: true},
{name: "failed without a reason",
conds: []batchv1.JobCondition{cond(batchv1.JobFailed, corev1.ConditionTrue, "")},
state: OpFailed, reason: "Failed", finished: true},
{name: "complete but its log is gone",
conds: []batchv1.JobCondition{complete},
state: OpFailed, reason: "ResultUnavailable", finished: true},
{name: "complete with a result that does not parse",
conds: []batchv1.JobCondition{complete}, log: ResultPrefix + "{\n",
state: OpFailed, reason: "ResultUnavailable", finished: true},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
st := opState(asyncJob(tc.conds...), tc.log)
if st.ID != "0a" || st.Op != OpUnzip || st.Path != "maps/world.zip" || !st.Started.Equal(opCreated) {
t.Fatalf("identity = %q %q %q %v", st.ID, st.Op, st.Path, st.Started)
}
if st.State != tc.state || st.Reason != tc.reason {
t.Fatalf("state = %q reason %q, want %q reason %q", st.State, st.Reason, tc.state, tc.reason)
}
if st.Done != tc.done || (tc.done != 0 && st.Total != 100) {
t.Fatalf("progress = %d/%d, want %d/100", st.Done, st.Total, tc.done)
}
if want := map[bool]time.Time{true: opEnded}[tc.finished]; !st.Finished.Equal(want) {
t.Fatalf("finished = %v, want %v", st.Finished, want)
}
if (st.Result != nil) != tc.wantResult {
t.Fatalf("result = %+v, want one: %v", st.Result, tc.wantResult)
}
if st.Result != nil && (st.Result.Code != tc.code || st.Result.Files != tc.files) {
t.Fatalf("result = %+v, want code %q and %d files", st.Result, tc.code, tc.files)
}
})
}
}
// TestK8sRunnerOps checks what Ops lists: this server's background Jobs only,
// newest first and at most maxOps of them, with a log read for each started Pod
// and none for a Pod still waiting to run.
func TestK8sRunnerOps(t *testing.T) {
job := func(server, id string, age time.Duration, async bool) *batchv1.Job {
p := testParams(OpUnzip)
p.Server, p.OpID, p.Path, p.Async = server, id, "maps/"+id+".zip", async
j, err := FilesJob(p)
if err != nil {
t.Fatal(err)
}
j.CreationTimestamp = metav1.NewTime(opCreated.Add(-age))
return j
}
pod := func(j *batchv1.Job, phase corev1.PodPhase, age time.Duration) *corev1.Pod {
return &corev1.Pod{
ObjectMeta: metav1.ObjectMeta{
Name: fmt.Sprintf("%s-%d", j.Name, age), Namespace: "minecraft", Labels: j.Spec.Template.Labels,
CreationTimestamp: metav1.NewTime(opCreated.Add(-age)),
},
Status: corev1.PodStatus{Phase: phase},
}
}
var objs []runtime.Object
for i := range maxOps + 2 {
j := job("survival", fmt.Sprintf("%02d", i), time.Duration(i)*time.Minute, true)
objs = append(objs, j, pod(j, corev1.PodSucceeded, time.Duration(i)*time.Minute))
}
// The newest Job's newest Pod has not started, so its log is not read; the
// older Pod beside it is not the one Ops reports on.
newest := job("survival", "new", -time.Minute, true)
objs = append(objs, newest,
pod(newest, corev1.PodPending, -time.Minute), pod(newest, corev1.PodFailed, 0))
objs = append(objs,
job("creative", "other", -2*time.Minute, true),
job("survival", "sync", -3*time.Minute, false))
cs := fake.NewSimpleClientset(objs...)
ops, err := NewK8sRunner(cs).Ops(context.Background(), "minecraft", "survival")
if err != nil {
t.Fatalf("Ops: %v", err)
}
var ids []string
for _, op := range ops {
ids = append(ids, op.ID)
}
want := []string{"new", "00", "01", "02", "03", "04", "05", "06", "07", "08"}
if fmt.Sprint(ids) != fmt.Sprint(want) {
t.Fatalf("ops = %v, want %v", ids, want)
}
if ops[0].Path != "maps/new.zip" || ops[0].Op != OpUnzip {
t.Fatalf("newest op = %+v", ops[0])
}
logs := 0
for _, a := range cs.Actions() {
if a.GetVerb() == "get" && a.GetSubresource() == "log" {
logs++
opts := a.(k8stesting.GenericAction).GetValue().(*corev1.PodLogOptions)
if opts.Container != containerName || opts.TailLines == nil || *opts.TailLines != opLogLines {
t.Fatalf("log options = %+v", opts)
}
}
}
if logs != maxOps-1 {
t.Fatalf("read %d logs, want %d: one per listed op whose Pod has started", logs, maxOps-1)
}
}
// TestK8sRunnerOpsFailsLoudly checks a listing the cluster refused is an error,
// never an empty list that would read as nothing running.
func TestK8sRunnerOpsFailsLoudly(t *testing.T) {
for _, resource := range []string{"jobs", "pods"} {
t.Run(resource, func(t *testing.T) {
cs := fake.NewSimpleClientset()
cs.PrependReactor("list", resource, func(k8stesting.Action) (bool, runtime.Object, error) {
return true, nil, fmt.Errorf("forbidden")
})
ops, err := NewK8sRunner(cs).Ops(context.Background(), "minecraft", "survival")
if err == nil || ops != nil {
t.Fatalf("Ops = %v, %v; want the refusal", ops, err)
}
})
}
}
// TestK8sRunnerStart checks Start creates the rendered Job and nothing else,
// and refuses params the renderer refuses without touching the cluster.
func TestK8sRunnerStart(t *testing.T) {
cs := fake.NewSimpleClientset()
p := testParams(OpUnzip)
p.Path, p.Async = "maps/world.zip", true
if err := NewK8sRunner(cs).Start(context.Background(), p); err != nil {
t.Fatalf("Start: %v", err)
}
want, _ := FilesJob(p)
got, err := cs.BatchV1().Jobs("minecraft").Get(context.Background(), want.Name, metav1.GetOptions{})
if err != nil {
t.Fatalf("the Job was not created: %v", err)
}
if got.Labels[LabelAsync] != "true" || got.Annotations[AnnotationPath] != "maps/world.zip" {
t.Fatalf("created %+v", got.ObjectMeta)
}
if n := len(cs.Actions()); n != 2 { // the create, and this test's get
t.Fatalf("%d calls to the cluster, want the one create", n-1)
}
cs = fake.NewSimpleClientset()
p.Op = OpRead
if err := NewK8sRunner(cs).Start(context.Background(), p); err == nil || len(cs.Actions()) != 0 {
t.Fatalf("err %v after %d calls, want a refusal before any", err, len(cs.Actions()))
}
cs = fake.NewSimpleClientset()
cs.PrependReactor("create", "jobs", func(k8stesting.Action) (bool, runtime.Object, error) {
return true, nil, fmt.Errorf("quota exceeded")
})
p.Op = OpUnzip
if err := NewK8sRunner(cs).Start(context.Background(), p); err == nil {
t.Fatal("a refused create must be an error")
}
}
+30 -8
View File
@@ -491,17 +491,39 @@ func TestUpload(t *testing.T) {
}
})
t.Run("over the cap is too_large and never fetched; at the cap is fetched", func(t *testing.T) {
t.Run("more than the volume has free is no_space and never fetched; exactly the free room is fetched", func(t *testing.T) {
root, _ := worldRoot(t)
stubStatfs(t, uint64(len(jar)), 1<<30)
src := &fakeSource{body: jar}
res, err := send(t, root, "big.jar", src.upload(MaxUploadBytes+1, ""), false)
if err != nil || res.Code != CodeTooLarge || src.opened != 0 {
t.Fatalf("result = %+v, %v, opened %d", res, err, src.opened)
res, err := send(t, root, "big.jar", src.upload(int64(len(jar))+1, ""), false)
if err != nil || res.Code != CodeNoSpace || res.Need != int64(len(jar))+1 || res.Avail != int64(len(jar)) || src.opened != 0 {
t.Fatalf("result = %+v, %v, opened %d; want no_space with need %d, avail %d", res, err, src.opened, len(jar)+1, len(jar))
}
// At the cap the size passes and the transfer starts; this source then
// comes up short, which is a broken transfer rather than a refusal.
if _, err := send(t, root, "big.jar", src.upload(MaxUploadBytes, ""), false); err == nil || src.opened != 1 {
t.Fatalf("at the cap: err = %v, opened %d; want a fetch", err, src.opened)
assertAbsent(t, filepath.Join(root, "big.jar"))
if res, err := send(t, root, "big.jar", whole(src), false); err != nil || res.Code != "" || src.opened != 1 {
t.Fatalf("at the free room: result = %+v, %v, opened %d; want it landed", res, err, src.opened)
}
})
// The editor used to stop at MaxUploadBytes; a file sent in parts is bounded
// by the volume alone.
t.Run("past the single-request limit is fetched", func(t *testing.T) {
root, _ := worldRoot(t)
stubStatfs(t, 1<<40, 1<<41)
src := &fakeSource{body: jar}
// This source then comes up short, which is a broken transfer, not a refusal.
if _, err := send(t, root, "big.jar", src.upload(MaxUploadBytes+1, ""), false); err == nil || src.opened != 1 {
t.Fatalf("err = %v, opened %d; want a fetch", err, src.opened)
}
})
t.Run("progress hears every byte", func(t *testing.T) {
root, _ := worldRoot(t)
var last, calls, total int64
res := exec(t, root, Request{Op: OpUpload, Path: "x.jar", Upload: whole(&fakeSource{body: jar}),
Progress: func(done, all int64) { calls++; last, total = done, all }})
if res.Code != "" || calls == 0 || last != int64(len(jar)) || total != int64(len(jar)) {
t.Fatalf("result = %+v; progress calls %d, last %d of %d; want the whole %d", res, calls, last, total, len(jar))
}
})
+61
View File
@@ -0,0 +1,61 @@
package fileedit
import (
"bufio"
"encoding/json"
"fmt"
"io"
"strings"
"time"
)
// ProgressPrefix marks a progress line: how many of an upload's or an unzip's
// bytes are in so far, as JSON. Those two run as Jobs felis-api does not wait on,
// and the panel shows how far one has got by reading the latest such line from
// the tail of the Pod's log. Like ResultPrefix it is found by its marker, since
// the log is stdout and stderr merged.
const ProgressPrefix = "FELIS-FILES-PROGRESS: "
// Progress is one progress line.
type Progress struct {
Done int64 `json:"done"`
Total int64 `json:"total"`
}
// ThrottledProgress returns a progress func that prints to w at most once per
// every, plus the first call (the last print starts at the zero time) and the
// one that reaches the total, so the log grows by a line a second however fast
// the bytes move and still ends on the true final count. A write error is
// dropped: progress is a courtesy, and the result line that follows is what
// felis-api acts on.
func ThrottledProgress(w io.Writer, every time.Duration, now func() time.Time) func(done, total int64) {
var last time.Time
return func(done, total int64) {
t := now()
if done < total && t.Sub(last) < every {
return
}
last = t
b, _ := json.Marshal(Progress{Done: done, Total: total})
fmt.Fprintf(w, "%s%s\n", ProgressPrefix, b)
}
}
// lastProgress finds the latest well-formed progress line in a log.
func lastProgress(log string) (Progress, bool) {
var p Progress
found := false
sc := bufio.NewScanner(strings.NewReader(log))
sc.Buffer(make([]byte, 0, 4096), maxLogBytes)
for sc.Scan() {
rest, ok := strings.CutPrefix(sc.Text(), ProgressPrefix)
if !ok {
continue
}
var q Progress
if json.Unmarshal([]byte(rest), &q) == nil {
p, found = q, true
}
}
return p, found
}
+42
View File
@@ -0,0 +1,42 @@
package fileedit
import (
"bytes"
"testing"
"time"
)
func TestThrottledProgress(t *testing.T) {
var out bytes.Buffer
clock := time.Unix(1000, 0)
progress := ThrottledProgress(&out, time.Second, func() time.Time { return clock })
progress(0, 100) // first call: printed
progress(10, 100) // same instant: dropped
clock = clock.Add(999 * time.Millisecond)
progress(20, 100) // not a second yet: dropped
clock = clock.Add(time.Millisecond)
progress(30, 100) // a second on: printed
progress(100, 100) // the total, however soon: printed
want := ProgressPrefix + `{"done":0,"total":100}` + "\n" +
ProgressPrefix + `{"done":30,"total":100}` + "\n" +
ProgressPrefix + `{"done":100,"total":100}` + "\n"
if out.String() != want {
t.Fatalf("printed:\n%s\nwant:\n%s", out.String(), want)
}
}
func TestLastProgress(t *testing.T) {
log := "noise\n" +
ProgressPrefix + `{"done":1,"total":9}` + "\n" +
"a runtime warning on stderr\n" +
ProgressPrefix + `{"done":5,"total":9}` + "\n" +
ProgressPrefix + `{"done":` + "\n" // cut off mid-line by the tail
p, ok := lastProgress(log)
if !ok || p != (Progress{Done: 5, Total: 9}) {
t.Fatalf("lastProgress = %+v, %v; want {5 9}, true", p, ok)
}
if _, ok := lastProgress("no marker here\n"); ok {
t.Fatal("a log without a progress line reported one")
}
}
+328
View File
@@ -0,0 +1,328 @@
package fileedit
import (
"crypto/sha256"
"crypto/subtle"
"encoding"
"errors"
"fmt"
"hash"
"io"
"os"
"time"
)
// A file too big for one request body arrives as a session: Begin declares its
// size and where it goes, Append adds one part at a time in order, and Seal
// hands the finished file to the Job that lands it, which fetches it through
// Open like any other staged upload. The Cloudflare edge refuses bodies over
// 100 MB, so a part is at most PartBytes; the file itself has no ceiling but the
// room on the staging disk, and Begin reserves all of it up front, so an upload
// that starts is one the disk can finish.
//
// Every call names the user and the server the session was begun for, and a
// session answers no one else: an id that is someone else's reads as unknown.
//
// A part that fails midway (the connection dropped, the edge cut it off) is
// rolled back to where it started, so the session's length is always the resume
// point. A sealed session stays until it has been served whole once (Served), so
// a Job that failed before it had every byte can be started again without the
// file being sent again; one left idle for SessionIdle is dropped (Expire).
// PartBytes is the largest part Append takes, matching the modpack upload's
// parts (submit.DefaultPartMaxBytes).
const PartBytes = 32 << 20
// SessionIdle is how long a session may sit untouched before Expire drops it:
// long enough to resume after a lost connection or a laptop lid, short enough
// that an abandoned upload gives its room back the same day.
const SessionIdle = 6 * time.Hour
// MaxSessionsPerUser bounds the sessions one user holds open at once. Each
// reserves its whole size on the staging disk, so without a bound one user
// could reserve the disk out from under everyone for SessionIdle.
const MaxSessionsPerUser = 4
var (
// ErrTooManySessions is a Begin by a user who already holds
// MaxSessionsPerUser sessions.
ErrTooManySessions = errors.New("fileedit: too many uploads in progress")
// ErrUploadBusy is a call on a session another request is still appending
// to. Parts go one at a time.
ErrUploadBusy = errors.New("fileedit: another request is still writing this upload")
// ErrPartTooLarge is a part over PartBytes, or one that runs past the size
// the session was begun with.
ErrPartTooLarge = errors.New("fileedit: the part is too large")
// ErrUploadIncomplete is a Seal before every byte has arrived.
ErrUploadIncomplete = errors.New("fileedit: the upload has not finished arriving")
)
// OffsetError is a part that does not start where the session ends. Received is
// where it does end, so the client resumes from there.
type OffsetError struct{ Received int64 }
func (e *OffsetError) Error() string {
return fmt.Sprintf("fileedit: the upload holds %d bytes; send the part that starts there", e.Received)
}
// Session is where one session stands.
type Session struct {
ID string
Path string
Size int64
Received int64
}
type session struct {
user, server, path string
file string
size, received int64
h hash.Hash
busy bool
touched time.Time
// armed is set by Seal with the digest of the token it minted and cleared by
// the Open that spends it.
armed bool
tokenHash [sha256.Size]byte
}
func (s *Stage) now() time.Time {
if s.Now != nil {
return s.Now()
}
return time.Now()
}
// Begin opens a session for a file of size bytes that will land at path on
// server, reserving room for all of it.
func (s *Stage) Begin(user, server, path string, size int64) (Session, error) {
if size < 0 {
return Session{}, fmt.Errorf("fileedit: an upload of %d bytes", size)
}
id, err := randomHex(16)
if err != nil {
return Session{}, fmt.Errorf("fileedit: generate an upload id: %w", err)
}
if err := os.MkdirAll(s.Dir, 0o700); err != nil {
return Session{}, fmt.Errorf("fileedit: create the upload stage: %w", err)
}
if err := s.reserve(size); err != nil {
return Session{}, err
}
f, err := os.CreateTemp(s.Dir, "session-*")
if err != nil {
s.unreserve(size)
return Session{}, fmt.Errorf("fileedit: stage the upload: %w", err)
}
f.Close()
s.mu.Lock()
defer s.mu.Unlock()
n := 0
for _, ss := range s.sessions {
if ss.user == user {
n++
}
}
if n >= MaxSessionsPerUser {
s.reserved -= size
os.Remove(f.Name())
return Session{}, fmt.Errorf("%w: finish or cancel one of your %d uploads first", ErrTooManySessions, n)
}
if s.sessions == nil {
s.sessions = map[string]*session{}
}
s.sessions[id] = &session{
user: user, server: server, path: path, file: f.Name(),
size: size, h: sha256.New(), touched: s.now(),
}
return Session{ID: id, Path: path, Size: size}, nil
}
// lookup finds the caller's session. s.mu must be held.
func (s *Stage) lookup(user, server, id string) (*session, error) {
ss, ok := s.sessions[id]
if !ok || ss.user != user || ss.server != server {
return nil, ErrNotStaged
}
return ss, nil
}
func (ss *session) view(id string) Session {
return Session{ID: id, Path: ss.path, Size: ss.size, Received: ss.received}
}
// Status reports where the caller's session stands.
func (s *Stage) Status(user, server, id string) (Session, error) {
s.mu.Lock()
defer s.mu.Unlock()
ss, err := s.lookup(user, server, id)
if err != nil {
return Session{}, err
}
return ss.view(id), nil
}
// Append adds the n bytes of body at offset, which must be where the session
// ends. body must end right after them (an HTTP body of that Content-Length
// does). On any failure the session is left as it was before the call.
func (s *Stage) Append(user, server, id string, offset int64, body io.Reader, n int64) (Session, error) {
s.mu.Lock()
ss, err := s.lookup(user, server, id)
switch {
case err != nil:
case ss.busy:
err = ErrUploadBusy
case offset != ss.received:
err = &OffsetError{Received: ss.received}
case n < 0 || n > PartBytes || n > ss.size-ss.received:
err = fmt.Errorf("%w: %d bytes at %d of a %d-byte upload; parts are at most %d bytes",
ErrPartTooLarge, n, offset, ss.size, PartBytes)
}
if err != nil {
var view Session
if ss != nil {
view = ss.view(id)
}
s.mu.Unlock()
return view, err
}
ss.busy = true
s.mu.Unlock()
// The session is ours until busy is cleared, so the file and the hash are
// touched without the lock. The hash's state is kept to undo a failed part.
before, err := ss.h.(encoding.BinaryMarshaler).MarshalBinary()
if err == nil {
err = appendPart(ss.file, offset, body, n, ss.h)
if err != nil {
_ = os.Truncate(ss.file, offset)
_ = ss.h.(encoding.BinaryUnmarshaler).UnmarshalBinary(before)
}
}
s.mu.Lock()
defer s.mu.Unlock()
ss.busy = false
ss.touched = s.now()
if err != nil {
return ss.view(id), err
}
ss.received += n
s.reserved -= n
return ss.view(id), nil
}
// appendPart writes exactly n bytes of body at offset in the file named file,
// feeding them to h as well.
func appendPart(file string, offset int64, body io.Reader, n int64, h hash.Hash) error {
f, err := os.OpenFile(file, os.O_WRONLY, 0)
if err != nil {
return fmt.Errorf("fileedit: open the staged upload: %w", err)
}
src := &bodyReader{r: body}
// One byte past n, so the read that finds the end happens here.
got, copyErr := io.Copy(io.MultiWriter(io.NewOffsetWriter(f, offset), h), io.LimitReader(src, n+1))
closeErr := f.Close()
return stageFailure(src.err, copyErr, closeErr, got, n)
}
// Seal ends the caller's session and arms it for one fetch: the Staged it
// returns carries a fresh token, and any token an earlier Seal minted stops
// working. Every byte must have arrived.
func (s *Stage) Seal(user, server, id string) (Staged, error) {
s.mu.Lock()
defer s.mu.Unlock()
ss, err := s.lookup(user, server, id)
if err != nil {
return Staged{}, err
}
if ss.busy {
return Staged{}, ErrUploadBusy
}
if ss.received != ss.size {
return Staged{}, fmt.Errorf("%w: %d of %d bytes are here", ErrUploadIncomplete, ss.received, ss.size)
}
st, tokenHash, err := newHandle(ss.h, ss.size)
if err != nil {
return Staged{}, err
}
st.ID = id
ss.armed, ss.tokenHash = true, tokenHash
ss.touched = s.now()
return st, nil
}
// openSession is Open for a sealed session. s.mu must be held; ok is false when
// id names no session.
func (s *Stage) openSession(id string, sum [sha256.Size]byte) (path string, size int64, ok bool, err error) {
ss, found := s.sessions[id]
if !found {
return "", 0, false, nil
}
if !ss.armed || subtle.ConstantTimeCompare(sum[:], ss.tokenHash[:]) != 1 {
return "", 0, true, ErrNotStaged
}
ss.armed = false
ss.touched = s.now()
return ss.file, ss.size, true, nil
}
// Served tells the stage the session id was sent whole to the Job that opened
// it, and deletes it: its bytes are on the Job's side now. An id that names no
// session (an upload staged by Put, which its own release deletes) is ignored.
func (s *Stage) Served(id string) {
s.mu.Lock()
ss, ok := s.sessions[id]
if ok {
delete(s.sessions, id)
}
s.mu.Unlock()
if ok {
os.Remove(ss.file)
}
}
// Drop cancels the caller's session and deletes what it holds.
func (s *Stage) Drop(user, server, id string) error {
s.mu.Lock()
ss, err := s.lookup(user, server, id)
if err == nil && ss.busy {
err = ErrUploadBusy
}
if err != nil {
s.mu.Unlock()
return err
}
s.dropLocked(id, ss)
s.mu.Unlock()
os.Remove(ss.file)
return nil
}
// dropLocked forgets a session and gives back the room it still had reserved.
// s.mu must be held; the caller deletes the file.
func (s *Stage) dropLocked(id string, ss *session) {
delete(s.sessions, id)
s.reserved -= ss.size - ss.received
}
// Expire drops every session untouched for SessionIdle, sealed or not, and
// reports how many it dropped. A session a part is arriving for is never idle.
func (s *Stage) Expire() int {
cutoff := s.now().Add(-SessionIdle)
var files []string
s.mu.Lock()
for id, ss := range s.sessions {
if !ss.busy && ss.touched.Before(cutoff) {
s.dropLocked(id, ss)
files = append(files, ss.file)
}
}
s.mu.Unlock()
for _, f := range files {
os.Remove(f)
}
return len(files)
}
+525
View File
@@ -0,0 +1,525 @@
package fileedit
import (
"errors"
"io"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
// diskStage is a stage on a disk of total bytes whose free space is what the
// files in it leave of free: statfs sees parts land, as a real disk would.
func diskStage(t *testing.T, free, total uint64, minFree float64) *Stage {
t.Helper()
s := &Stage{Dir: filepath.Join(t.TempDir(), "stage"), MinFree: minFree}
prev := statfs
statfs = func(string) (uint64, uint64, error) {
var used uint64
des, _ := os.ReadDir(s.Dir)
for _, de := range des {
if info, err := de.Info(); err == nil {
used += uint64(info.Size())
}
}
return free - used, total, nil
}
t.Cleanup(func() { statfs = prev })
return s
}
func appendString(s *Stage, user, server, id string, offset int64, part string) (Session, error) {
return s.Append(user, server, id, offset, strings.NewReader(part), int64(len(part)))
}
func readStaged(t *testing.T, s *Stage, id, token string) string {
t.Helper()
f, size, err := s.Open(id, token)
if err != nil {
t.Fatalf("Open: %v", err)
}
defer f.Close()
b, err := io.ReadAll(f)
if err != nil {
t.Fatal(err)
}
if size != int64(len(b)) {
t.Fatalf("Open said %d bytes and served %d", size, len(b))
}
return string(b)
}
// TestSessionArrivesInParts: parts land in order, Seal hands the Job a token for
// exactly those bytes, and Served deletes them.
func TestSessionArrivesInParts(t *testing.T) {
s := roomyStage(t)
const whole = "PK\x03\x04 first part, second part"
sess, err := s.Begin("u1", "survival", "plugins/big.jar", int64(len(whole)))
if err != nil {
t.Fatalf("Begin: %v", err)
}
if !hexID.MatchString(sess.ID) || sess != (Session{ID: sess.ID, Path: "plugins/big.jar", Size: int64(len(whole))}) {
t.Fatalf("Begin = %+v", sess)
}
got, err := appendString(s, "u1", "survival", sess.ID, 0, whole[:16])
if err != nil || got.Received != 16 || got.Size != int64(len(whole)) {
t.Fatalf("first part: %+v, %v", got, err)
}
if at, err := s.Status("u1", "survival", sess.ID); err != nil || at.Received != 16 || at.Path != "plugins/big.jar" {
t.Fatalf("Status = %+v, %v", at, err)
}
if got, err = appendString(s, "u1", "survival", sess.ID, 16, whole[16:]); err != nil || got.Received != int64(len(whole)) {
t.Fatalf("second part: %+v, %v", got, err)
}
st, err := s.Seal("u1", "survival", sess.ID)
if err != nil {
t.Fatalf("Seal: %v", err)
}
if st.ID != sess.ID || !hexToken.MatchString(st.Token) || st.Size != int64(len(whole)) || st.SHA256 != digest([]byte(whole)) {
t.Fatalf("Seal = %+v, want the digest of %q", st, whole)
}
if body := readStaged(t, s, st.ID, st.Token); body != whole {
t.Fatalf("served %q, want %q", body, whole)
}
if _, _, err := s.Open(st.ID, st.Token); !errors.Is(err, ErrNotStaged) {
t.Fatalf("second Open with the same token: err = %v, want ErrNotStaged", err)
}
s.Served(st.ID)
if names := stagedNames(t, s); len(names) != 0 {
t.Fatalf("after Served: %v", names)
}
if _, err := s.Status("u1", "survival", sess.ID); !errors.Is(err, ErrNotStaged) {
t.Fatalf("Status after Served: err = %v, want ErrNotStaged", err)
}
}
// A session answers only the user and the server it was begun for.
func TestSessionAnswersItsOwnerOnly(t *testing.T) {
s := roomyStage(t)
sess, err := s.Begin("u1", "survival", "a.zip", 4)
if err != nil {
t.Fatal(err)
}
for name, who := range map[string][2]string{
"another user": {"u2", "survival"},
"another server": {"u1", "creative"},
} {
if _, err := s.Status(who[0], who[1], sess.ID); !errors.Is(err, ErrNotStaged) {
t.Errorf("%s: Status err = %v", name, err)
}
if _, err := appendString(s, who[0], who[1], sess.ID, 0, "abcd"); !errors.Is(err, ErrNotStaged) {
t.Errorf("%s: Append err = %v", name, err)
}
if _, err := s.Seal(who[0], who[1], sess.ID); !errors.Is(err, ErrNotStaged) {
t.Errorf("%s: Seal err = %v", name, err)
}
if err := s.Drop(who[0], who[1], sess.ID); !errors.Is(err, ErrNotStaged) {
t.Errorf("%s: Drop err = %v", name, err)
}
}
if _, err := s.Status("u1", "survival", strings.Repeat("0", 32)); !errors.Is(err, ErrNotStaged) {
t.Errorf("unknown id: err = %v", err)
}
if at, err := s.Status("u1", "survival", sess.ID); err != nil || at.Received != 0 {
t.Fatalf("the owner's session after the others tried: %+v, %v", at, err)
}
}
func TestSessionRefusesAPartThatDoesNotFit(t *testing.T) {
s := roomyStage(t)
sess, err := s.Begin("u1", "survival", "a.zip", 6)
if err != nil {
t.Fatal(err)
}
if _, err := appendString(s, "u1", "survival", sess.ID, 0, "abc"); err != nil {
t.Fatal(err)
}
var off *OffsetError
for _, offset := range []int64{0, 2, 4} {
at, err := appendString(s, "u1", "survival", sess.ID, offset, "d")
if !errors.As(err, &off) || off.Received != 3 || at.Received != 3 {
t.Fatalf("offset %d: %+v, err = %v; want an OffsetError at 3", offset, at, err)
}
}
if _, err := appendString(s, "u1", "survival", sess.ID, 3, "defg"); !errors.Is(err, ErrPartTooLarge) {
t.Fatalf("past the declared size: err = %v, want ErrPartTooLarge", err)
}
if _, err := s.Append("u1", "survival", sess.ID, 3, strings.NewReader(""), -1); !errors.Is(err, ErrPartTooLarge) {
t.Fatalf("negative length: err = %v, want ErrPartTooLarge", err)
}
if at, err := appendString(s, "u1", "survival", sess.ID, 3, "def"); err != nil || at.Received != 6 {
t.Fatalf("the part that fits exactly: %+v, %v", at, err)
}
big, err := s.Begin("u1", "survival", "b.zip", PartBytes+2)
if err != nil {
t.Fatal(err)
}
if _, err := s.Append("u1", "survival", big.ID, 0, strings.NewReader(""), PartBytes+1); !errors.Is(err, ErrPartTooLarge) {
t.Fatalf("a part over PartBytes: err = %v, want ErrPartTooLarge", err)
}
}
// A part that breaks or runs long leaves the session as it was: the file is cut
// back and the digest forgets it, so the resent part makes the right file.
func TestSessionRollsBackAFailedPart(t *testing.T) {
for name, tc := range map[string]struct {
body io.Reader
short bool
}{
"breaks": {io.MultiReader(strings.NewReader("XY"), errReader{io.ErrUnexpectedEOF}), true},
"ends": {strings.NewReader("XY"), true},
"runs long": {strings.NewReader("XYZWV"), false},
} {
t.Run(name, func(t *testing.T) {
s := roomyStage(t)
sess, err := s.Begin("u1", "survival", "a.zip", 7)
if err != nil {
t.Fatal(err)
}
if _, err := appendString(s, "u1", "survival", sess.ID, 0, "abc"); err != nil {
t.Fatal(err)
}
at, err := s.Append("u1", "survival", sess.ID, 3, tc.body, 4)
if err == nil || errors.Is(err, ErrShortUpload) != tc.short || at.Received != 3 {
t.Fatalf("%+v, err = %v; want a failure at 3 (short = %v)", at, err, tc.short)
}
info, err := os.Stat(filepath.Join(s.Dir, stagedNames(t, s)[0]))
if err != nil || info.Size() != 3 {
t.Fatalf("staged file is %v bytes (%v), want it cut back to 3", info.Size(), err)
}
if _, err := appendString(s, "u1", "survival", sess.ID, 3, "defg"); err != nil {
t.Fatalf("resent part: %v", err)
}
st, err := s.Seal("u1", "survival", sess.ID)
if err != nil || st.SHA256 != digest([]byte("abcdefg")) {
t.Fatalf("Seal = %+v, %v; want the digest of abcdefg", st, err)
}
if body := readStaged(t, s, st.ID, st.Token); body != "abcdefg" {
t.Fatalf("served %q", body)
}
})
}
}
// While a part is arriving nothing else may touch the session, and it is never
// idle.
func TestSessionIsBusyWhileAPartArrives(t *testing.T) {
s := roomyStage(t)
now := time.Date(2026, 9, 28, 12, 0, 0, 0, time.UTC)
s.Now = func() time.Time { return now }
sess, err := s.Begin("u1", "survival", "a.zip", 4)
if err != nil {
t.Fatal(err)
}
pr, pw := io.Pipe()
done := make(chan error, 1)
go func() {
_, err := s.Append("u1", "survival", sess.ID, 0, pr, 4)
done <- err
}()
// The write returns once Append is copying, which is after it marked busy.
if _, err := pw.Write([]byte("ab")); err != nil {
t.Fatal(err)
}
if _, err := appendString(s, "u1", "survival", sess.ID, 0, "abcd"); !errors.Is(err, ErrUploadBusy) {
t.Errorf("a second part: err = %v, want ErrUploadBusy", err)
}
if _, err := s.Seal("u1", "survival", sess.ID); !errors.Is(err, ErrUploadBusy) {
t.Errorf("Seal: err = %v, want ErrUploadBusy", err)
}
if err := s.Drop("u1", "survival", sess.ID); !errors.Is(err, ErrUploadBusy) {
t.Errorf("Drop: err = %v, want ErrUploadBusy", err)
}
now = now.Add(SessionIdle + time.Hour)
if n := s.Expire(); n != 0 {
t.Errorf("Expire dropped %d sessions with a part arriving", n)
}
if _, err := pw.Write([]byte("cd")); err != nil {
t.Fatal(err)
}
pw.Close()
if err := <-done; err != nil {
t.Fatalf("the part in flight: %v", err)
}
if _, err := s.Seal("u1", "survival", sess.ID); err != nil {
t.Fatalf("Seal once the part is in: %v", err)
}
}
// Each Seal arms one fetch with a fresh token, so a Job that failed can be
// started again on the same bytes.
func TestSessionSealArmsOneFetch(t *testing.T) {
s := roomyStage(t)
sess, err := s.Begin("u1", "survival", "a.zip", 4)
if err != nil {
t.Fatal(err)
}
if _, err := appendString(s, "u1", "survival", sess.ID, 0, "abc"); err != nil {
t.Fatal(err)
}
if _, err := s.Seal("u1", "survival", sess.ID); !errors.Is(err, ErrUploadIncomplete) {
t.Fatalf("Seal at 3 of 4: err = %v, want ErrUploadIncomplete", err)
}
if _, _, err := s.Open(sess.ID, ""); !errors.Is(err, ErrNotStaged) {
t.Fatalf("Open before any Seal: err = %v, want ErrNotStaged", err)
}
if _, err := appendString(s, "u1", "survival", sess.ID, 3, "d"); err != nil {
t.Fatal(err)
}
first, err := s.Seal("u1", "survival", sess.ID)
if err != nil {
t.Fatal(err)
}
second, err := s.Seal("u1", "survival", sess.ID)
if err != nil {
t.Fatal(err)
}
if first.Token == second.Token || first.SHA256 != second.SHA256 {
t.Fatalf("two Seals: %+v and %+v; want fresh tokens for the same bytes", first, second)
}
if _, _, err := s.Open(sess.ID, first.Token); !errors.Is(err, ErrNotStaged) {
t.Fatalf("the replaced token: err = %v, want ErrNotStaged", err)
}
if _, _, err := s.Open(sess.ID, strings.Repeat("0", 64)); !errors.Is(err, ErrNotStaged) {
t.Fatalf("a wrong token: err = %v, want ErrNotStaged", err)
}
// Neither wrong token spent the armed one.
if body := readStaged(t, s, sess.ID, second.Token); body != "abcd" {
t.Fatalf("served %q", body)
}
if _, _, err := s.Open(sess.ID, second.Token); !errors.Is(err, ErrNotStaged) {
t.Fatalf("the spent token: err = %v, want ErrNotStaged", err)
}
// Opened but not served whole: the Job broke midway, and a new Seal serves
// the same bytes again.
third, err := s.Seal("u1", "survival", sess.ID)
if err != nil {
t.Fatal(err)
}
if body := readStaged(t, s, sess.ID, third.Token); body != "abcd" {
t.Fatalf("served %q after a new Seal", body)
}
}
// Served deletes only sessions: an upload staged by Put belongs to the release
// func Put returned.
func TestServedLeavesPutAlone(t *testing.T) {
s := roomyStage(t)
st, release, err := s.Put(strings.NewReader("abc"), 3)
if err != nil {
t.Fatal(err)
}
defer release()
s.Served(st.ID)
if body := readStaged(t, s, st.ID, st.Token); body != "abc" {
t.Fatalf("served %q", body)
}
}
// A session reserves its whole size when it begins, and gives back what it has
// not yet received when it is dropped or expires.
func TestSessionReservesItsSize(t *testing.T) {
t.Run("begin reserves the whole size", func(t *testing.T) {
s := diskStage(t, 1000, 1200, 0.5) // room for 400
if _, err := s.Begin("u1", "survival", "a.zip", 300); err != nil {
t.Fatal(err)
}
if _, err := s.Begin("u2", "survival", "b.zip", 101); !errors.Is(err, ErrStageFull) {
t.Fatalf("101 bytes beside a 300-byte session: err = %v, want ErrStageFull", err)
}
if _, err := s.Begin("u2", "survival", "b.zip", 100); err != nil {
t.Fatalf("100 bytes beside a 300-byte session: %v", err)
}
})
t.Run("a part moves its room from the reservation to the disk", func(t *testing.T) {
s := diskStage(t, 1000, 1200, 0.5)
sess, err := s.Begin("u1", "survival", "a.zip", 300)
if err != nil {
t.Fatal(err)
}
if _, err := appendString(s, "u1", "survival", sess.ID, 0, strings.Repeat("x", 200)); err != nil {
t.Fatal(err)
}
if _, err := s.Begin("u2", "survival", "b.zip", 101); !errors.Is(err, ErrStageFull) {
t.Fatalf("after a part landed: err = %v, want ErrStageFull", err)
}
if _, err := s.Begin("u2", "survival", "b.zip", 100); err != nil {
t.Fatalf("after a part landed: %v", err)
}
})
t.Run("drop gives it all back", func(t *testing.T) {
s := diskStage(t, 1000, 1200, 0.5)
sess, err := s.Begin("u1", "survival", "a.zip", 300)
if err != nil {
t.Fatal(err)
}
if _, err := appendString(s, "u1", "survival", sess.ID, 0, strings.Repeat("x", 200)); err != nil {
t.Fatal(err)
}
if err := s.Drop("u1", "survival", sess.ID); err != nil {
t.Fatal(err)
}
if names := stagedNames(t, s); len(names) != 0 {
t.Fatalf("after Drop: %v", names)
}
if _, err := s.Begin("u2", "survival", "b.zip", 400); err != nil {
t.Fatalf("after Drop: %v", err)
}
if _, err := s.Status("u1", "survival", sess.ID); !errors.Is(err, ErrNotStaged) {
t.Fatalf("Status after Drop: err = %v", err)
}
})
t.Run("expiry gives it all back", func(t *testing.T) {
s := diskStage(t, 1000, 1200, 0.5)
now := time.Date(2026, 9, 28, 12, 0, 0, 0, time.UTC)
s.Now = func() time.Time { return now }
sess, err := s.Begin("u1", "survival", "a.zip", 300)
if err != nil {
t.Fatal(err)
}
if _, err := appendString(s, "u1", "survival", sess.ID, 0, strings.Repeat("x", 200)); err != nil {
t.Fatal(err)
}
now = now.Add(SessionIdle + time.Nanosecond)
if n := s.Expire(); n != 1 {
t.Fatalf("Expire dropped %d, want 1", n)
}
if _, err := s.Begin("u2", "survival", "b.zip", 400); err != nil {
t.Fatalf("after Expire: %v", err)
}
})
}
func TestSessionsPerUserAreBounded(t *testing.T) {
s := diskStage(t, 1000, 1200, 0.5)
var ids []string
for i := range MaxSessionsPerUser {
sess, err := s.Begin("u1", "survival", "a.zip", 10)
if err != nil {
t.Fatalf("session %d: %v", i+1, err)
}
ids = append(ids, sess.ID)
}
if _, err := s.Begin("u1", "creative", "a.zip", 10); !errors.Is(err, ErrTooManySessions) {
t.Fatalf("one more on another server: err = %v, want ErrTooManySessions", err)
}
// The refused Begin kept neither a file nor its reservation: room for 400
// less the four sessions' 40.
if names := stagedNames(t, s); len(names) != MaxSessionsPerUser {
t.Fatalf("staged %d files, want %d", len(names), MaxSessionsPerUser)
}
if _, err := s.Begin("u2", "survival", "b.zip", 360); err != nil {
t.Fatalf("another user: %v", err)
}
if err := s.Drop("u1", "survival", ids[0]); err != nil {
t.Fatal(err)
}
if _, err := s.Begin("u1", "survival", "a.zip", 10); err != nil {
t.Fatalf("after dropping one: %v", err)
}
// With room reserved, a negative size would wrap the reservation sum round
// to a small number and pass the room check.
if _, err := s.Begin("u3", "survival", "a.zip", -1); err == nil || errors.Is(err, ErrStageFull) {
t.Fatalf("a negative size: err = %v, want it refused for being negative", err)
}
}
// Expire drops what has sat untouched for longer than SessionIdle, sealed or
// not, and a part keeps a session alive.
func TestSessionExpiry(t *testing.T) {
s := roomyStage(t)
start := time.Date(2026, 9, 28, 12, 0, 0, 0, time.UTC)
now := start
s.Now = func() time.Time { return now }
idle, err := s.Begin("u1", "survival", "idle.zip", 2)
if err != nil {
t.Fatal(err)
}
sealed, err := s.Begin("u1", "survival", "sealed.zip", 1)
if err != nil {
t.Fatal(err)
}
if _, err := appendString(s, "u1", "survival", sealed.ID, 0, "x"); err != nil {
t.Fatal(err)
}
if _, err := s.Seal("u1", "survival", sealed.ID); err != nil {
t.Fatal(err)
}
active, err := s.Begin("u1", "survival", "active.zip", 2)
if err != nil {
t.Fatal(err)
}
now = start.Add(time.Hour)
if _, err := appendString(s, "u1", "survival", active.ID, 0, "a"); err != nil {
t.Fatal(err)
}
now = start.Add(SessionIdle)
if n := s.Expire(); n != 0 {
t.Fatalf("at exactly SessionIdle Expire dropped %d", n)
}
now = start.Add(SessionIdle + time.Minute)
if n := s.Expire(); n != 2 {
t.Fatalf("Expire dropped %d, want the idle and the sealed one", n)
}
for _, id := range []string{idle.ID, sealed.ID} {
if _, err := s.Status("u1", "survival", id); !errors.Is(err, ErrNotStaged) {
t.Errorf("%s survived Expire: %v", id, err)
}
}
if at, err := s.Status("u1", "survival", active.ID); err != nil || at.Received != 1 {
t.Fatalf("the session a part touched: %+v, %v", at, err)
}
if names := stagedNames(t, s); len(names) != 1 {
t.Fatalf("files left: %v, want the active session's", names)
}
// A Seal, and the Job's Open, each start the idle clock again: a Job begun
// on an upload that sat for hours still finds it there.
for _, tc := range []struct {
name string
// sealAt and openAt are how long after the last part the Seal and the
// Job's Open come; a negative openAt is no Open.
sealAt, openAt time.Duration
}{
{"seal", 4 * time.Hour, -1},
{"open", 0, 4 * time.Hour},
} {
begun := now
sess, err := s.Begin("u1", "survival", tc.name+".zip", 1)
if err != nil {
t.Fatal(err)
}
if _, err := appendString(s, "u1", "survival", sess.ID, 0, "x"); err != nil {
t.Fatal(err)
}
now = begun.Add(tc.sealAt)
st, err := s.Seal("u1", "survival", sess.ID)
if err != nil {
t.Fatal(err)
}
if tc.openAt >= 0 {
now = begun.Add(tc.openAt)
readStaged(t, s, sess.ID, st.Token)
}
now = begun.Add(SessionIdle + time.Minute)
s.Expire()
if _, err := s.Status("u1", "survival", sess.ID); err != nil {
t.Errorf("%s: a session touched 4h after its last part expired 6h after it: %v", tc.name, err)
}
if err := s.Drop("u1", "survival", sess.ID); err != nil {
t.Fatal(err)
}
}
}
+38 -20
View File
@@ -12,6 +12,7 @@ import (
"os"
"sync"
"syscall"
"time"
)
// Stage holds uploads between the request that brought them and the Job that
@@ -25,7 +26,9 @@ import (
// Job carries in its environment. Only a digest of the token is kept, compared in
// constant time, and the first successful Open spends it: the Job never retries,
// so a second Open could only be someone else. The release func Put returns
// deletes the file once the Job has answered, whatever it answered.
// deletes the file once the Job has answered, whatever it answered. A file too
// big for one request arrives in parts instead (session.go) and is fetched the
// same way.
//
// Nothing here outlives the process: the index is in memory, so Sweep empties
// Dir at startup of whatever a previous process left behind.
@@ -38,8 +41,12 @@ type Stage struct {
// the submission store shares.
MinFree float64
// Now is the clock sessions are aged by (time.Now when nil).
Now func() time.Time
mu sync.Mutex
items map[string]*stagedFile
sessions map[string]*session
reserved int64
}
@@ -172,23 +179,27 @@ func stageFailure(readErr, copyErr, closeErr error, n, size int64) error {
// newHandle mints the id and token for a staged upload whose bytes h hashed.
func newHandle(h hash.Hash, size int64) (Staged, [sha256.Size]byte, error) {
var id [16]byte
var token [32]byte
if _, err := rand.Read(id[:]); err != nil {
id, err := randomHex(16)
if err != nil {
return Staged{}, [sha256.Size]byte{}, fmt.Errorf("fileedit: generate an upload id: %w", err)
}
if _, err := rand.Read(token[:]); err != nil {
token, err := randomHex(32)
if err != nil {
return Staged{}, [sha256.Size]byte{}, fmt.Errorf("fileedit: generate an upload token: %w", err)
}
st := Staged{
ID: hex.EncodeToString(id[:]),
Token: hex.EncodeToString(token[:]),
SHA256: hex.EncodeToString(h.Sum(nil)),
Size: size,
}
st := Staged{ID: id, Token: token, SHA256: hex.EncodeToString(h.Sum(nil)), Size: size}
return st, sha256.Sum256([]byte(st.Token)), nil
}
// randomHex is n random bytes in hex.
func randomHex(n int) (string, error) {
b := make([]byte, n)
if _, err := rand.Read(b); err != nil {
return "", err
}
return hex.EncodeToString(b), nil
}
// reserve admits an upload of size bytes if the disk keeps MinFree free after it
// and after every upload still being written. Those have not reached the disk
// yet, so statfs alone would let two of them through on room for one.
@@ -219,23 +230,30 @@ func (s *Stage) unreserve(size int64) {
s.mu.Unlock()
}
// Open spends a staged upload's token and returns its file and size. Any
// mismatch is ErrNotStaged.
// Open spends a staged upload's token, or a sealed session's (Seal), and returns
// its file and size. Any mismatch is ErrNotStaged.
func (s *Stage) Open(id, token string) (*os.File, int64, error) {
sum := sha256.Sum256([]byte(token))
s.mu.Lock()
it, ok := s.items[id]
if !ok || it.used || subtle.ConstantTimeCompare(sum[:], it.tokenHash[:]) != 1 {
s.mu.Unlock()
return nil, 0, ErrNotStaged
name, size, found, err := s.openSession(id, sum)
if !found {
it, ok := s.items[id]
if !ok || it.used || subtle.ConstantTimeCompare(sum[:], it.tokenHash[:]) != 1 {
err = ErrNotStaged
} else {
it.used = true
name, size = it.path, it.size
}
}
it.used = true
s.mu.Unlock()
f, err := os.Open(it.path)
if err != nil {
return nil, 0, err
}
f, err := os.Open(name)
if err != nil {
return nil, 0, fmt.Errorf("fileedit: open the staged upload: %w", err)
}
return f, it.size, nil
return f, size, nil
}
// bodyReader remembers the body's own read error, so Put can tell a client that
+536
View File
@@ -0,0 +1,536 @@
package fileedit
import (
"archive/zip"
"crypto/rand"
"encoding/hex"
"errors"
"fmt"
"io"
"io/fs"
"math"
"os"
"path"
"sort"
"strings"
"unicode/utf8"
"golang.org/x/text/encoding/simplifiedchinese"
)
// The codes an unzip refuses an archive with. Each names what is wrong with the
// ARCHIVE, so the panel can say "this zip is broken" rather than "your path is
// wrong"; Result.Entry names the entry.
const (
// CodeArchiveInvalid is a file that is not a zip, a damaged one, one whose
// entry bytes disagree with their header (size or CRC), or one with nothing in it.
CodeArchiveInvalid = "archive_invalid"
// CodeArchiveUnsafe is an entry naming a path outside the folder it is
// extracted into ("../", "/etc/x", "C:\x"), or a device, pipe or socket.
CodeArchiveUnsafe = "archive_unsafe"
// CodeArchiveSymlink is an entry that is a symbolic link. A link extracted into
// the world could point anywhere, and every later op would have to reason about
// it, so an archive carrying one is refused whole.
CodeArchiveSymlink = "archive_symlink"
// CodeTypeConflict is an archive with a file where the server has a folder, a
// folder where it has a file, or anything where it has a link. Overwrite never
// resolves it: replacing a folder with a file would delete the folder.
CodeTypeConflict = "type_conflict"
)
// MaxConflicts bounds Result.Conflicts, keeping the result line bounded when an
// archive would replace a whole world; ConflictCount still says how many there
// are.
const MaxConflicts = 200
// unzipEntryOverhead is what the space check adds per entry for the inode and
// directory block it takes beyond its bytes.
const unzipEntryOverhead = 4096
// unzipTempPrefix names the working folder an unzip extracts into. It sits in
// the destination folder, so every move out of it is a rename on one volume.
const unzipTempPrefix = ".felis-unzip-"
// unzip extracts the .zip at name into the folder holding it.
//
// It is all or nothing. Every check that can refuse the archive runs before a
// byte is written: entry names, entry types, what is already on the server, and
// the room on the volume. The entries are then extracted into a working folder
// beside the destination, and only once every one of them has been written and
// verified are they renamed into place. A failure at any point before that
// leaves the destination exactly as it was. The renames themselves are journaled
// and undone in reverse if one fails, so a replaced file comes back. Only a Job
// killed in the middle of the renames — a few milliseconds for thousands of
// files — can leave the archive half applied.
//
// An existing file the archive would replace is a conflict: without overwrite the
// unzip lists them (CodeExists, Conflicts) and changes nothing; with it they are
// replaced. Folders merge. A file where the server has a folder, or the reverse,
// is CodeTypeConflict whatever overwrite says.
//
// The only size bound is the volume. Each entry's declared size is summed and
// checked against the free space up front, and archive/zip itself refuses an
// entry whose bytes run past its declared size or fail its CRC, so an archive
// that lies about its sizes (a zip bomb) stops at the first lying entry and
// nothing it wrote survives.
func unzip(r *os.Root, rootPath, name string, overwrite bool, progress func(done, total int64)) Result {
name = path.Clean(name)
if !strings.EqualFold(path.Ext(name), ".zip") {
return Result{Code: CodeBadPath, Error: fmt.Sprintf("%s is not a .zip archive", name)}
}
f, err := r.Open(name)
if err != nil {
return failure(err, name)
}
defer f.Close()
info, err := f.Stat()
if err != nil {
return failure(err, name)
}
if !info.Mode().IsRegular() {
return Result{Code: CodeBadPath, Error: fmt.Sprintf("%s is not a file", name)}
}
// ErrInsecurePath comes back WITH a usable reader, and only under
// GODEBUG=zipinsecurepath=0; planUnzip does that check itself, for every
// entry, whatever the setting.
zr, err := zip.NewReader(f, info.Size())
if err != nil && !errors.Is(err, zip.ErrInsecurePath) {
return Result{Code: CodeArchiveInvalid, Error: fmt.Sprintf("%s is not a readable zip archive: %v", name, err)}
}
p, res := planUnzip(zr.File)
if res.Code != "" {
return res
}
dest := path.Dir(name)
present, replaced, res := checkTargets(r, dest, p)
if res.Code != "" {
return res
}
if len(replaced) > 0 && !overwrite {
list := make([]string, 0, len(replaced))
for n := range replaced {
list = append(list, path.Join(dest, n))
}
sort.Strings(list)
count := len(list)
if count > MaxConflicts {
list = list[:MaxConflicts]
}
return Result{Code: CodeExists, Conflicts: list, ConflictCount: count, Error: fmt.Sprintf(
"%d files in the archive already exist on the server; extract again with overwrite to replace them", count)}
}
// A working folder left by an unzip that was killed holds only a copy, and
// clearing it first gives its room back to the check below.
sweepUnzipTemps(r, dest)
need := p.bytes + int64(len(p.files)+len(p.dirs))*unzipEntryOverhead
if avail, _, err := statfs(rootPath); err == nil && uint64(need) > avail {
free := int64(math.MaxInt64)
if avail < math.MaxInt64 {
free = int64(avail)
}
return Result{Code: CodeNoSpace, Need: need, Avail: free, Error: fmt.Sprintf(
"extracting %s needs %d bytes and the server's volume has %d free; nothing was changed", name, need, free)}
}
var suffix [6]byte
if _, err := rand.Read(suffix[:]); err != nil {
return Result{Code: CodeBadPath, Error: fmt.Sprintf("generate a temporary name: %v", err)}
}
tmp := path.Join(dest, unzipTempPrefix+hex.EncodeToString(suffix[:]))
staged, old := path.Join(tmp, "new"), path.Join(tmp, "old")
for _, d := range []string{tmp, staged, old} {
if err := r.Mkdir(d, 0o700); err != nil {
_ = r.RemoveAll(tmp)
return unzipWriteFailure(err, dest)
}
}
// After a success tmp holds only the files the archive replaced; after a
// failure, everything the archive wrote. Either way it goes.
defer func() { _ = r.RemoveAll(tmp) }()
if res := extractAll(r, staged, p, progress); res.Code != "" {
return res
}
if res := placeAll(r, dest, staged, old, p, present, replaced); res.Code != "" {
return res
}
return Result{Files: len(p.files), Bytes: p.bytes}
}
// unzipPlan is an archive's entries once every one has passed planUnzip. Names
// are cleaned, slash-separated and relative to the destination folder.
type unzipPlan struct {
files []zipFile
// dirs is every folder the archive makes, named in it or implied by a file
// inside it, sorted so a folder comes before everything in it.
dirs []string
isDir map[string]bool
bytes int64 // the declared size of every file, summed
}
type zipFile struct {
f *zip.File
name string
mode fs.FileMode
}
// planUnzip checks every entry's name and type and works out what the archive
// makes. Nothing about the server is consulted yet.
func planUnzip(entries []*zip.File) (unzipPlan, Result) {
p := unzipPlan{isDir: map[string]bool{}}
byName := map[string]bool{}
for _, f := range entries {
raw := entryName(f)
name, ok := cleanEntry(raw)
if !ok {
return p, Result{Code: CodeArchiveUnsafe, Entry: raw, Error: fmt.Sprintf(
"%s leads outside the folder it would be extracted into", raw)}
}
// macOS's Finder adds __MACOSX/ to every zip it makes: resource forks that
// mean nothing on the server.
if name == "__MACOSX" || strings.HasPrefix(name, "__MACOSX/") {
continue
}
mode := f.Mode()
isDir := mode.IsDir() || strings.HasSuffix(raw, "/")
switch {
case mode&fs.ModeSymlink != 0:
return p, Result{Code: CodeArchiveSymlink, Entry: raw, Error: fmt.Sprintf(
"%s is a symbolic link; an archive containing links is not extracted", raw)}
case isDir:
if name != "." {
p.isDir[name] = true
}
continue
case !mode.IsRegular():
return p, Result{Code: CodeArchiveUnsafe, Entry: raw, Error: fmt.Sprintf(
"%s is not a regular file", raw)}
case name == ".":
return p, Result{Code: CodeArchiveUnsafe, Entry: raw, Error: fmt.Sprintf(
"%s names the destination folder itself", raw)}
case byName[name]:
return p, Result{Code: CodeArchiveInvalid, Entry: raw, Error: fmt.Sprintf(
"%s appears in the archive twice", raw)}
case f.UncompressedSize64 > uint64(math.MaxInt64-p.bytes):
return p, Result{Code: CodeArchiveInvalid, Entry: raw, Error: fmt.Sprintf(
"%s declares an impossible size", raw)}
}
byName[name] = true
p.bytes += int64(f.UncompressedSize64)
// Anything the archive marks executable (a start.sh) stays executable;
// every other permission is the server's usual.
perm := fs.FileMode(0o644)
if mode.Perm()&0o111 != 0 {
perm = 0o755
}
p.files = append(p.files, zipFile{f: f, name: name, mode: perm})
}
for _, zf := range p.files {
// cleanEntry already refused a rooted name; stopping at "/" as well keeps
// this loop finite should that check ever move.
for d := path.Dir(zf.name); d != "." && d != "/"; d = path.Dir(d) {
p.isDir[d] = true
}
}
for _, zf := range p.files {
if p.isDir[zf.name] {
return p, Result{Code: CodeArchiveInvalid, Entry: zf.name, Error: fmt.Sprintf(
"%s is both a file and a folder in the archive", zf.name)}
}
}
if len(p.files) == 0 && len(p.isDir) == 0 {
return p, Result{Code: CodeArchiveInvalid, Error: "the archive has nothing to extract"}
}
for d := range p.isDir {
p.dirs = append(p.dirs, d)
}
// A folder's name is a prefix of everything in it, and a prefix sorts first.
sort.Strings(p.dirs)
sort.Slice(p.files, func(i, j int) bool { return p.files[i].name < p.files[j].name })
return p, Result{}
}
// entryName is an entry's name as its maker meant it. A zip made on Chinese
// Windows stores names in the system code page (GBK) without the UTF-8 flag; a
// name that is not valid UTF-8 is decoded as GB18030, GBK's superset. The flag
// alone is no guide: archive/zip reports NonUTF8 for every name made without it,
// which includes plain ASCII and macOS's UTF-8. Decoding comes before the
// backslash below because a GBK trail byte can be 0x5C.
func entryName(f *zip.File) string {
name := f.Name
if !utf8.ValidString(name) {
if s, err := simplifiedchinese.GB18030.NewDecoder().String(name); err == nil {
name = s
}
}
return strings.ReplaceAll(name, `\`, "/")
}
// cleanEntry cleans an entry name and reports false for one that must not be
// extracted: a NUL, an absolute path, a drive letter, or a climb out of the
// destination. It judges the name as TEXT, which is sound here because nothing
// is resolved through it until checkTargets and extraction, and those go through
// os.Root, which would refuse an escape anyway.
func cleanEntry(raw string) (string, bool) {
if raw == "" || strings.ContainsRune(raw, 0) || strings.HasPrefix(raw, "/") {
return "", false
}
if len(raw) >= 2 && raw[1] == ':' && (raw[0]|0x20) >= 'a' && (raw[0]|0x20) <= 'z' {
return "", false
}
name := path.Clean(raw)
if name == ".." || strings.HasPrefix(name, "../") {
return "", false
}
return name, true
}
// checkTargets looks at what the server already has where the archive lands.
// present is the archive's folders that exist on the server (they merge);
// replaced is its files that do (conflicts). A folder the server lacks cannot
// hold anything, so its contents are not looked up.
func checkTargets(r *os.Root, dest string, p unzipPlan) (present, replaced map[string]bool, res Result) {
present, replaced = map[string]bool{}, map[string]bool{}
absent := func(name string) bool {
parent := path.Dir(name)
return parent != "." && !present[parent]
}
for _, d := range p.dirs {
if absent(d) {
continue
}
at := path.Join(dest, d)
info, err := r.Lstat(at)
switch {
case errors.Is(err, fs.ErrNotExist):
case err != nil:
return nil, nil, failure(err, at)
case info.Mode()&fs.ModeSymlink != 0:
return nil, nil, typeConflict(at, "is a link on the server, and the archive has a folder there")
case info.IsDir():
present[d] = true
default:
return nil, nil, typeConflict(at, "is a file on the server, and the archive has a folder there")
}
}
for _, zf := range p.files {
if absent(zf.name) {
continue
}
at := path.Join(dest, zf.name)
info, err := r.Lstat(at)
switch {
case errors.Is(err, fs.ErrNotExist):
case err != nil:
return nil, nil, failure(err, at)
case info.Mode().IsRegular():
replaced[zf.name] = true
case info.IsDir():
return nil, nil, typeConflict(at, "is a folder on the server, and the archive has a file there")
default:
return nil, nil, typeConflict(at, "is a link or special file on the server, and the archive has a file there")
}
}
return present, replaced, Result{}
}
func typeConflict(at, why string) Result {
return Result{Code: CodeTypeConflict, Entry: at, Error: fmt.Sprintf("%s %s; nothing was changed", at, why)}
}
// extractAll writes every folder and file of p under staged, handed to the game
// uid, and syncs each file.
func extractAll(r *os.Root, staged string, p unzipPlan, progress func(done, total int64)) Result {
for _, d := range p.dirs {
at := path.Join(staged, d)
if err := r.Mkdir(at, 0o755); err != nil {
return unzipWriteFailure(err, d)
}
_ = ownWritten(r, at)
}
var done int64
count := func(n int) {
done += int64(n)
if progress != nil {
progress(done, p.bytes)
}
}
for _, zf := range p.files {
if res := extractOne(r, path.Join(staged, zf.name), zf, count); res.Code != "" {
return res
}
}
for _, d := range p.dirs {
syncDir(r, path.Join(staged, d))
}
syncDir(r, staged)
return Result{}
}
func extractOne(r *os.Root, at string, zf zipFile, count func(int)) Result {
invalid := func(err error) Result {
return Result{Code: CodeArchiveInvalid, Entry: zf.name, Error: fmt.Sprintf(
"%s in the archive is damaged: %v; nothing was changed", zf.name, err)}
}
src, err := zf.f.Open()
if err != nil {
return invalid(err)
}
defer src.Close()
w, err := r.OpenFile(at, os.O_WRONLY|os.O_CREATE|os.O_EXCL, zf.mode)
if err != nil {
return unzipWriteFailure(err, zf.name)
}
if err := w.Chmod(zf.mode); err != nil {
w.Close()
return unzipWriteFailure(err, zf.name)
}
if _, err := io.Copy(countingWriter{w, count}, archiveReader{src}); err != nil {
w.Close()
var ae *archiveError
if errors.As(err, &ae) {
return invalid(ae.err)
}
return unzipWriteFailure(err, zf.name)
}
if err := syncWritten(w); err != nil {
w.Close()
return unzipWriteFailure(err, zf.name)
}
if err := w.Close(); err != nil {
return unzipWriteFailure(err, zf.name)
}
_ = ownWritten(r, at)
return Result{}
}
// placeAll renames the extracted tree into dest. A folder the server lacks moves
// whole; one it has is descended into. A file it has is first moved aside into
// old, so undoing the journal puts it back.
func placeAll(r *os.Root, dest, staged, old string, p unzipPlan, present, replaced map[string]bool) Result {
kids := map[string][]string{}
for _, d := range p.dirs {
kids[path.Dir(d)] = append(kids[path.Dir(d)], d)
}
for _, zf := range p.files {
kids[path.Dir(zf.name)] = append(kids[path.Dir(zf.name)], zf.name)
}
type move struct{ from, to string }
var journal []move
mv := func(from, to string) error {
if err := renameEntry(r, from, to); err != nil {
return err
}
journal = append(journal, move{from, to})
return nil
}
var place func(dir string) error
place = func(dir string) error {
for _, c := range kids[dir] {
src, dst := path.Join(staged, c), path.Join(dest, c)
switch {
case p.isDir[c] && present[c]:
if err := place(c); err != nil {
return err
}
case replaced[c]:
aside := path.Join(old, c)
if err := r.MkdirAll(path.Dir(aside), 0o700); err != nil {
return err
}
if err := mv(dst, aside); err != nil {
return err
}
if err := mv(src, dst); err != nil {
return err
}
default:
if err := mv(src, dst); err != nil {
return err
}
}
}
return nil
}
if err := place("."); err != nil {
for i := len(journal) - 1; i >= 0; i-- {
_ = r.Rename(journal[i].to, journal[i].from)
}
return unzipWriteFailure(err, dest)
}
syncDir(r, dest)
for d := range present {
syncDir(r, path.Join(dest, d))
}
return Result{}
}
// renameEntry is the rename placeAll moves with. A var so a test can fail one
// part-way through and watch the journal undo the rest.
var renameEntry = func(r *os.Root, from, to string) error { return r.Rename(from, to) }
// sweepUnzipTemps removes working folders a killed unzip left in dir.
func sweepUnzipTemps(r *os.Root, dir string) {
d, err := r.Open(dir)
if err != nil {
return
}
names, _ := d.Readdirnames(-1)
d.Close()
for _, n := range names {
if isUnzipTemp(n) {
_ = r.RemoveAll(path.Join(dir, n))
}
}
}
func isUnzipTemp(name string) bool {
suffix, ok := strings.CutPrefix(name, unzipTempPrefix)
if !ok || len(suffix) != 12 {
return false
}
_, err := hex.DecodeString(suffix)
return err == nil
}
// unzipWriteFailure is writeFailure worded for an unzip, which by then has
// changed nothing whatever the step that failed.
func unzipWriteFailure(err error, name string) Result {
res := writeFailure(err, name)
if res.Code == CodeNoSpace {
res.Error = "the server's volume filled up while extracting; nothing was changed"
}
return res
}
// archiveError marks a failure reading an entry's bytes out of the archive, so
// extractOne can tell a damaged archive from the volume failing underneath.
type archiveError struct{ err error }
func (e *archiveError) Error() string { return "read archive: " + e.err.Error() }
func (e *archiveError) Unwrap() error { return e.err }
type archiveReader struct{ r io.Reader }
func (a archiveReader) Read(p []byte) (int, error) {
n, err := a.r.Read(p)
if err != nil && err != io.EOF {
err = &archiveError{err}
}
return n, err
}
// countingWriter reports each write's length to add.
type countingWriter struct {
w io.Writer
add func(int)
}
func (c countingWriter) Write(p []byte) (int, error) {
n, err := c.w.Write(p)
c.add(n)
return n, err
}
+539
View File
@@ -0,0 +1,539 @@
package fileedit
import (
"archive/zip"
"bytes"
"errors"
"fmt"
"hash/crc32"
"io"
"io/fs"
"os"
"path/filepath"
"reflect"
"strings"
"syscall"
"testing"
"golang.org/x/text/encoding/simplifiedchinese"
)
// zent is one entry of a test archive: a deflated file, or a folder when the name
// ends in "/".
type zent struct {
name string
body string
mode fs.FileMode // set on the header when non-zero
flat bool // written without the UTF-8 flag, as old Windows tools do
// raw writes body stored as-is under a header declaring size and crc, so a
// test can build an archive whose entries lie about themselves.
raw bool
size uint64
crc uint32
}
func file(name, body string) zent { return zent{name: name, body: body} }
// lie is an entry declaring size bytes while holding body.
func lie(name, body string, size uint64) zent {
return zent{name: name, body: body, raw: true, size: size, crc: crc32.ChecksumIEEE([]byte(body))}
}
func writeZip(t *testing.T, at string, entries ...zent) {
t.Helper()
var buf bytes.Buffer
w := zip.NewWriter(&buf)
for _, e := range entries {
fh := &zip.FileHeader{Name: e.name, Method: zip.Deflate, NonUTF8: e.flat}
if e.mode != 0 {
fh.SetMode(e.mode)
}
var dst io.Writer
var err error
if e.raw {
fh.Method = zip.Store
fh.CRC32, fh.CompressedSize64, fh.UncompressedSize64 = e.crc, uint64(len(e.body)), e.size
dst, err = w.CreateRaw(fh)
} else {
dst, err = w.CreateHeader(fh)
}
if err != nil {
t.Fatalf("zip entry %q: %v", e.name, err)
}
if _, err := io.WriteString(dst, e.body); err != nil {
t.Fatalf("zip entry %q: %v", e.name, err)
}
}
if err := w.Close(); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(at, buf.Bytes(), 0o644); err != nil {
t.Fatal(err)
}
}
// tree is everything under dir: each file's content, "<dir>" for a folder and
// "-> target" for a link. Comparing two trees is how a test says "nothing
// changed", working folders included.
func tree(t *testing.T, dir string) map[string]string {
t.Helper()
out := map[string]string{}
err := filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
rel, _ := filepath.Rel(dir, p)
switch {
case rel == ".":
case d.Type()&fs.ModeSymlink != 0:
target, err := os.Readlink(p)
if err != nil {
return err
}
out[rel] = "-> " + target
case d.IsDir():
out[rel] = "<dir>"
default:
b, err := os.ReadFile(p)
if err != nil {
return err
}
out[rel] = string(b)
}
return nil
})
if err != nil {
t.Fatal(err)
}
return out
}
func assertSameTree(t *testing.T, before, after map[string]string) {
t.Helper()
if !reflect.DeepEqual(before, after) {
t.Fatalf("the tree changed:\nbefore %v\nafter %v", before, after)
}
}
func unzipAt(t *testing.T, root, name string, overwrite bool) Result {
t.Helper()
return exec(t, root, Request{Op: OpUnzip, Path: name, Overwrite: overwrite})
}
func TestUnzip(t *testing.T) {
t.Run("extracts files and folders into the folder holding the archive", func(t *testing.T) {
root, _ := worldRoot(t)
if err := os.Mkdir(filepath.Join(root, "plugins"), 0o755); err != nil {
t.Fatal(err)
}
writeZip(t, filepath.Join(root, "plugins", "pack.zip"),
zent{name: "Essentials/"},
file("Essentials/config.yml", "locale: zh\n"),
zent{name: "empty/"},
file("./readme.txt", "hi"),
file(`win\sub\a.txt`, "from windows"),
)
var owned []string
prev := ownWritten
ownWritten = func(_ *os.Root, name string) error { owned = append(owned, name); return nil }
defer func() { ownWritten = prev }()
res := unzipAt(t, root, "plugins/pack.zip", false)
if res.Code != "" || res.Files != 3 || res.Bytes != int64(len("locale: zh\n")+len("hi")+len("from windows")) {
t.Fatalf("result = %+v", res)
}
got := tree(t, filepath.Join(root, "plugins"))
delete(got, "pack.zip")
want := map[string]string{
"Essentials": "<dir>", "Essentials/config.yml": "locale: zh\n", "empty": "<dir>",
"readme.txt": "hi", "win": "<dir>", "win/sub": "<dir>", "win/sub/a.txt": "from windows",
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("plugins/ = %v\nwant %v", got, want)
}
if info, _ := os.Stat(filepath.Join(root, "plugins", "readme.txt")); info.Mode().Perm() != 0o644 {
t.Fatalf("mode = %v, want 0644", info.Mode().Perm())
}
// Every folder and file is handed to the game uid, while still in the
// working folder: 4 folders and 3 files.
if len(owned) != 7 {
t.Fatalf("handed to the game uid: %v, want 7", owned)
}
for _, o := range owned {
if !strings.HasPrefix(o, "plugins/"+unzipTempPrefix) {
t.Fatalf("%s was chowned outside the working folder", o)
}
}
})
t.Run("an archive named in capitals extracts", func(t *testing.T) {
root, _ := worldRoot(t)
writeZip(t, filepath.Join(root, "PACK.ZIP"), file("a.txt", "a"))
if res := unzipAt(t, root, "PACK.ZIP", false); res.Code != "" || res.Files != 1 {
t.Fatalf("result = %+v", res)
}
if got := mustRead(t, filepath.Join(root, "a.txt")); got != "a" {
t.Fatalf("a.txt = %q", got)
}
})
t.Run("__MACOSX is left out", func(t *testing.T) {
root, _ := worldRoot(t)
writeZip(t, filepath.Join(root, "mac.zip"), file("a.txt", "a"), zent{name: "__MACOSX/"}, file("__MACOSX/._a.txt", "fork"))
if res := unzipAt(t, root, "mac.zip", false); res.Code != "" || res.Files != 1 {
t.Fatalf("result = %+v", res)
}
assertAbsent(t, filepath.Join(root, "__MACOSX"))
})
t.Run("a name made on Chinese Windows is decoded from GBK; UTF-8 without the flag is kept", func(t *testing.T) {
root, _ := worldRoot(t)
gbk, err := simplifiedchinese.GBK.NewEncoder().String("存档/说明.txt")
if err != nil {
t.Fatal(err)
}
writeZip(t, filepath.Join(root, "cn.zip"),
zent{name: gbk, body: "中文", flat: true},
zent{name: "macOS名字.txt", body: "utf8", flat: true},
)
if res := unzipAt(t, root, "cn.zip", false); res.Code != "" || res.Files != 2 {
t.Fatalf("result = %+v", res)
}
if got := mustRead(t, filepath.Join(root, "存档", "说明.txt")); got != "中文" {
t.Fatalf("存档/说明.txt = %q", got)
}
if got := mustRead(t, filepath.Join(root, "macOS名字.txt")); got != "utf8" {
t.Fatalf("macOS名字.txt = %q", got)
}
})
t.Run("an executable entry stays executable; every other file is 0644", func(t *testing.T) {
root, _ := worldRoot(t)
writeZip(t, filepath.Join(root, "sh.zip"),
zent{name: "start.sh", body: "#!/bin/sh\n", mode: 0o755},
zent{name: "secret.txt", body: "x", mode: 0o600},
)
if res := unzipAt(t, root, "sh.zip", false); res.Code != "" {
t.Fatalf("result = %+v", res)
}
for name, want := range map[string]fs.FileMode{"start.sh": 0o755, "secret.txt": 0o644} {
if info, _ := os.Stat(filepath.Join(root, name)); info.Mode().Perm() != want {
t.Errorf("%s mode = %v, want %v", name, info.Mode().Perm(), want)
}
}
})
t.Run("a name leading outside is archive_unsafe and nothing is written", func(t *testing.T) {
for raw, entry := range map[string]string{
"../evil.txt": "../evil.txt",
"a/../../evil.txt": "a/../../evil.txt",
`..\evil.txt`: "../evil.txt",
"/evil.txt": "/evil.txt",
"C:/evil.txt": "C:/evil.txt",
`c:\evil.txt`: "c:/evil.txt",
"nul\x00.txt": "nul\x00.txt",
"a/..": "a/..",
} {
t.Run(entry, func(t *testing.T) {
root, outside := worldRoot(t)
writeZip(t, filepath.Join(root, "bad.zip"), file("ok.txt", "ok"), file(raw, "evil"))
before := tree(t, root)
res := unzipAt(t, root, "bad.zip", true)
if res.Code != CodeArchiveUnsafe || res.Entry != entry {
t.Fatalf("result = %+v; want archive_unsafe naming %q", res, entry)
}
assertSameTree(t, before, tree(t, root))
assertAbsent(t, filepath.Join(outside, "evil.txt"))
})
}
})
t.Run("a link entry is archive_symlink and a pipe is archive_unsafe", func(t *testing.T) {
for _, tc := range []struct {
mode fs.FileMode
code string
}{
{fs.ModeSymlink | 0o777, CodeArchiveSymlink},
{fs.ModeNamedPipe | 0o644, CodeArchiveUnsafe},
} {
root, _ := worldRoot(t)
writeZip(t, filepath.Join(root, "odd.zip"),
file("ok.txt", "ok"), zent{name: "odd", body: "../../outside/secret.txt", mode: tc.mode})
before := tree(t, root)
res := unzipAt(t, root, "odd.zip", true)
if res.Code != tc.code || res.Entry != "odd" {
t.Fatalf("%v: result = %+v; want %s naming odd", tc.mode, res, tc.code)
}
assertSameTree(t, before, tree(t, root))
}
})
t.Run("a damaged or senseless archive is archive_invalid and nothing is written", func(t *testing.T) {
for name, tc := range map[string]struct {
entries []zent
entry string
}{
"bytes past the declared size": {[]zent{file("ok.txt", "ok"), lie("lie.txt", "0123456789", 3)}, "lie.txt"},
"bytes short of the declared size": {[]zent{file("ok.txt", "ok"), lie("lie.txt", "0123456789", 20)}, "lie.txt"},
"a wrong checksum": {[]zent{file("ok.txt", "ok"),
{name: "crc.txt", body: "0123456789", raw: true, size: 10, crc: crc32.ChecksumIEEE([]byte("0123456789")) + 1}}, "crc.txt"},
"the same file twice": {[]zent{file("a.txt", "1"), file("a.txt", "2")}, "a.txt"},
"a file and a folder at once": {[]zent{file("a", "1"), file("a/b.txt", "2")}, "a"},
"nothing in it": {nil, ""},
"nothing but __MACOSX": {[]zent{file("__MACOSX/._a", "fork")}, ""},
} {
t.Run(name, func(t *testing.T) {
root, _ := worldRoot(t)
writeZip(t, filepath.Join(root, "bad.zip"), tc.entries...)
before := tree(t, root)
res := unzipAt(t, root, "bad.zip", true)
if res.Code != CodeArchiveInvalid || res.Entry != tc.entry {
t.Fatalf("result = %+v; want archive_invalid naming %q", res, tc.entry)
}
assertSameTree(t, before, tree(t, root))
})
}
t.Run("not a zip at all", func(t *testing.T) {
root, _ := worldRoot(t)
if err := os.WriteFile(filepath.Join(root, "fake.zip"), []byte("hello"), 0o644); err != nil {
t.Fatal(err)
}
if res := unzipAt(t, root, "fake.zip", false); res.Code != CodeArchiveInvalid {
t.Fatalf("result = %+v; want archive_invalid", res)
}
})
})
t.Run("the archive's own path is checked", func(t *testing.T) {
root, outside := worldRoot(t)
if err := os.Mkdir(filepath.Join(root, "dir.zip"), 0o755); err != nil {
t.Fatal(err)
}
writeZip(t, filepath.Join(outside, "x.zip"), file("evil.txt", "evil"))
symlink(t, outside, filepath.Join(root, "escape-link"))
for name, code := range map[string]string{
"server.properties": CodeBadPath,
"missing.zip": CodeNotFound,
"dir.zip": CodeBadPath,
"../outside/x.zip": CodeBadPath,
"escape-link/x.zip": CodeBadPath,
"config/../../x.zip": CodeBadPath,
} {
if res := unzipAt(t, root, name, true); res.Code != code {
t.Errorf("%s: result = %+v; want %s", name, res, code)
}
}
assertAbsent(t, filepath.Join(root, "evil.txt"))
assertAbsent(t, filepath.Join(outside, "evil.txt"))
})
t.Run("files already there are listed and nothing changes without overwrite", func(t *testing.T) {
root, _ := worldRoot(t)
writeZip(t, filepath.Join(root, "pack.zip"),
file("server.properties", "motd=new\n"), file("config/paper.yml", "verbose: true\n"),
file("config/new.yml", "new"), file("fresh/x.txt", "x"))
before := tree(t, root)
res := unzipAt(t, root, "pack.zip", false)
if res.Code != CodeExists || res.ConflictCount != 2 ||
!reflect.DeepEqual(res.Conflicts, []string{"config/paper.yml", "server.properties"}) {
t.Fatalf("result = %+v; want exists listing config/paper.yml and server.properties", res)
}
assertSameTree(t, before, tree(t, root))
})
t.Run("conflicts are named from the server's root when the archive sits in a folder", func(t *testing.T) {
root, _ := worldRoot(t)
writeZip(t, filepath.Join(root, "config", "pack.zip"), file("paper.yml", "verbose: true\n"))
res := unzipAt(t, root, "config/pack.zip", false)
if res.Code != CodeExists || !reflect.DeepEqual(res.Conflicts, []string{"config/paper.yml"}) {
t.Fatalf("result = %+v", res)
}
})
t.Run("the list stops at MaxConflicts and the count does not", func(t *testing.T) {
root, _ := worldRoot(t)
var entries []zent
for i := range MaxConflicts + 1 {
name := fmt.Sprintf("f%03d.txt", i)
if err := os.WriteFile(filepath.Join(root, name), []byte("old"), 0o644); err != nil {
t.Fatal(err)
}
entries = append(entries, file(name, "new"))
}
writeZip(t, filepath.Join(root, "many.zip"), entries...)
res := unzipAt(t, root, "many.zip", false)
if res.Code != CodeExists || res.ConflictCount != MaxConflicts+1 || len(res.Conflicts) != MaxConflicts ||
res.Conflicts[0] != "f000.txt" || res.Conflicts[MaxConflicts-1] != "f199.txt" {
t.Fatalf("code %q, count %d, %d listed (%v … %v)", res.Code, res.ConflictCount, len(res.Conflicts),
res.Conflicts[:1], res.Conflicts[len(res.Conflicts)-1:])
}
})
t.Run("overwrite replaces files, merges folders and keeps everything else", func(t *testing.T) {
root, _ := worldRoot(t)
if err := os.WriteFile(filepath.Join(root, "config", "keep.yml"), []byte("keep"), 0o644); err != nil {
t.Fatal(err)
}
writeZip(t, filepath.Join(root, "pack.zip"),
file("server.properties", "motd=new\n"), file("config/paper.yml", "verbose: true\n"),
file("config/new.yml", "new"), file("fresh/x.txt", "x"))
res := unzipAt(t, root, "pack.zip", true)
if res.Code != "" || res.Files != 4 {
t.Fatalf("result = %+v", res)
}
got := tree(t, root)
delete(got, "pack.zip")
want := map[string]string{
"server.properties": "motd=new\n", "config": "<dir>", "config/paper.yml": "verbose: true\n",
"config/keep.yml": "keep", "config/new.yml": "new", "fresh": "<dir>", "fresh/x.txt": "x",
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("world = %v\nwant %v", got, want)
}
})
t.Run("a file where the server has a folder, the reverse, or a link is type_conflict even with overwrite", func(t *testing.T) {
for name, tc := range map[string]struct {
entry string
want string
}{
"folder over a file": {"server.properties/x.txt", "server.properties"},
"file over a folder": {"config", "config"},
"folder over a link": {"escape-link/evil.txt", "escape-link"},
"file over a link": {"planted.txt", "planted.txt"},
} {
t.Run(name, func(t *testing.T) {
root, outside := worldRoot(t)
symlink(t, outside, filepath.Join(root, "escape-link"))
symlink(t, "server.properties", filepath.Join(root, "planted.txt"))
writeZip(t, filepath.Join(root, "pack.zip"), file("ok.txt", "ok"), file(tc.entry, "evil"))
before := tree(t, root)
res := unzipAt(t, root, "pack.zip", true)
if res.Code != CodeTypeConflict || res.Entry != tc.want {
t.Fatalf("result = %+v; want type_conflict naming %s", res, tc.want)
}
assertSameTree(t, before, tree(t, root))
assertAbsent(t, filepath.Join(outside, "evil.txt"))
})
}
})
t.Run("more than the volume has free is no_space before anything is written", func(t *testing.T) {
root, _ := worldRoot(t)
writeZip(t, filepath.Join(root, "pack.zip"), file("d/a.txt", "0123456789"), file("d/b.txt", "0123456789"))
need := int64(20 + 3*unzipEntryOverhead) // two files and the folder d
stubStatfs(t, uint64(need-1), 1<<30)
before := tree(t, root)
res := unzipAt(t, root, "pack.zip", false)
if res.Code != CodeNoSpace || res.Need != need || res.Avail != need-1 {
t.Fatalf("result = %+v; want no_space with need %d, avail %d", res, need, need-1)
}
assertSameTree(t, before, tree(t, root))
stubStatfs(t, uint64(need), 1<<30)
if res := unzipAt(t, root, "pack.zip", false); res.Code != "" {
t.Fatalf("with exactly enough room: %+v", res)
}
})
t.Run("the volume filling up mid-way is no_space and changes nothing", func(t *testing.T) {
root, _ := worldRoot(t)
writeZip(t, filepath.Join(root, "pack.zip"), file("a.txt", "a"), file("server.properties", "motd=new\n"))
calls := 0
prev := syncWritten
syncWritten = func(f *os.File) error {
if calls++; calls == 2 {
return syscall.ENOSPC
}
return f.Sync()
}
defer func() { syncWritten = prev }()
before := tree(t, root)
if res := unzipAt(t, root, "pack.zip", true); res.Code != CodeNoSpace {
t.Fatalf("result = %+v; want no_space", res)
}
assertSameTree(t, before, tree(t, root))
})
// The renames are the one step that touches the server's files; one failing
// part-way must put back every file already moved, replaced ones included.
t.Run("a rename failing part-way is undone, whichever it is", func(t *testing.T) {
// config present: paper.yml aside + in, z.yml in; zeta in; server.properties aside + in.
const moves = 6
for failAt := 1; failAt <= moves; failAt++ {
root, _ := worldRoot(t)
writeZip(t, filepath.Join(root, "pack.zip"),
file("config/paper.yml", "verbose: true\n"), file("config/z.yml", "z"),
file("server.properties", "motd=new\n"), file("zeta/x.txt", "x"))
calls := 0
prev := renameEntry
renameEntry = func(r *os.Root, from, to string) error {
if calls++; calls == failAt {
return errors.New("injected rename failure")
}
return r.Rename(from, to)
}
before := tree(t, root)
res := unzipAt(t, root, "pack.zip", true)
renameEntry = prev
if res.Code == "" {
t.Fatalf("rename %d failing: result = %+v; want a failure", failAt, res)
}
assertSameTree(t, before, tree(t, root))
}
// And with none failing, the count above is the real number of moves.
root, _ := worldRoot(t)
writeZip(t, filepath.Join(root, "pack.zip"),
file("config/paper.yml", "verbose: true\n"), file("config/z.yml", "z"),
file("server.properties", "motd=new\n"), file("zeta/x.txt", "x"))
calls := 0
prev := renameEntry
renameEntry = func(r *os.Root, from, to string) error { calls++; return r.Rename(from, to) }
defer func() { renameEntry = prev }()
if res := unzipAt(t, root, "pack.zip", true); res.Code != "" || calls != moves {
t.Fatalf("result = %+v, %d moves; want success in %d", res, calls, moves)
}
})
t.Run("a working folder a killed unzip left is cleared; a lookalike is kept", func(t *testing.T) {
root, _ := worldRoot(t)
stale := filepath.Join(root, unzipTempPrefix+"0123456789ab")
if err := os.MkdirAll(filepath.Join(stale, "new"), 0o700); err != nil {
t.Fatal(err)
}
// Twelve characters that are not hex, and hex a byte short or long.
lookalikes := []string{"notahexname!", "0123456789", "0123456789abcd"}
for _, l := range lookalikes {
if err := os.Mkdir(filepath.Join(root, unzipTempPrefix+l), 0o755); err != nil {
t.Fatal(err)
}
}
writeZip(t, filepath.Join(root, "pack.zip"), file("a.txt", "a"))
if res := unzipAt(t, root, "pack.zip", false); res.Code != "" {
t.Fatalf("result = %+v", res)
}
assertAbsent(t, stale)
for _, l := range lookalikes {
if _, err := os.Stat(filepath.Join(root, unzipTempPrefix+l)); err != nil {
t.Fatalf("lookalike %q removed: %v", l, err)
}
}
})
t.Run("progress climbs to the total", func(t *testing.T) {
root, _ := worldRoot(t)
big := strings.Repeat("x", 100<<10)
writeZip(t, filepath.Join(root, "pack.zip"), file("a.bin", big), file("b.bin", big))
var seen []int64
var total int64
res := exec(t, root, Request{Op: OpUnzip, Path: "pack.zip",
Progress: func(done, all int64) { seen = append(seen, done); total = all }})
if res.Code != "" || total != int64(2*len(big)) || len(seen) < 2 || seen[len(seen)-1] != total {
t.Fatalf("result = %+v; progress %d calls ending at %v of %d", res, len(seen), seen[len(seen)-1:], total)
}
for i := 1; i < len(seen); i++ {
if seen[i] < seen[i-1] {
t.Fatalf("progress went backwards: %v", seen)
}
}
})
}