feat(files): 大文件分片上传、停服解压 zip 先列冲突再覆盖、文件和文件夹可下载;导出和下载不再带出 RCON 密码与转发密钥
This commit is contained in:
76 files changed
+10542
-569
No files matched your search
@@ -0,0 +1,215 @@
|
||||
package fileedit
|
||||
|
||||
import (
|
||||
"archive/zip"
|
||||
"bytes"
|
||||
"compress/flate"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path"
|
||||
)
|
||||
|
||||
// Content types of a download: a file goes out as its bytes, a folder as a zip.
|
||||
const (
|
||||
DownloadFileType = "application/octet-stream"
|
||||
DownloadZipType = "application/zip"
|
||||
)
|
||||
|
||||
// Download is one file or folder of a world on its way to the owner's browser,
|
||||
// run by the export Job (cmd/felis export --mode files). It passes the same
|
||||
// guards a read does (Guard): the forwarding-secret file never leaves, and
|
||||
// server.properties leaves with its RCON password redacted.
|
||||
type Download struct {
|
||||
// Size is a file download's exact length, or -1 for a folder, whose zip is
|
||||
// written as it streams.
|
||||
Size int64
|
||||
ContentType string
|
||||
// Skipped and Withheld count, once WriteTo has run, the entries a folder
|
||||
// download left out: links, devices and sockets, and guarded files.
|
||||
Skipped, Withheld int
|
||||
|
||||
root *os.Root
|
||||
name string
|
||||
file *os.File // a file download
|
||||
body []byte // a redacted file download
|
||||
guard Guard
|
||||
}
|
||||
|
||||
// OpenDownload opens name under rootPath for download. dir is what the caller
|
||||
// saw at name when it asked (the panel's listing): a download of a file that has
|
||||
// since become a folder, or the reverse, is refused rather than sent as the
|
||||
// other thing. The world root itself is refused; the world export sends that.
|
||||
func OpenDownload(rootPath, name string, dir bool) (*Download, error) {
|
||||
name = path.Clean(name)
|
||||
if name == "." || name == "/" || !fs.ValidPath(name) {
|
||||
return nil, fmt.Errorf("%s is not a file or folder inside the world", name)
|
||||
}
|
||||
r, err := os.OpenRoot(rootPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("open the world root: %w", err)
|
||||
}
|
||||
d := &Download{root: r, name: name, guard: NewGuard(r)}
|
||||
if err := d.open(dir); err != nil {
|
||||
r.Close()
|
||||
return nil, err
|
||||
}
|
||||
return d, nil
|
||||
}
|
||||
|
||||
func (d *Download) open(dir bool) error {
|
||||
info, err := d.root.Stat(d.name)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if info.IsDir() != dir {
|
||||
if info.IsDir() {
|
||||
return fmt.Errorf("%s is a folder now; reload the file list and download it again", d.name)
|
||||
}
|
||||
return fmt.Errorf("%s is not a folder now; reload the file list and download it again", d.name)
|
||||
}
|
||||
if dir {
|
||||
d.Size, d.ContentType = -1, DownloadZipType
|
||||
return nil
|
||||
}
|
||||
if !info.Mode().IsRegular() {
|
||||
return fmt.Errorf("%s is not a regular file", d.name)
|
||||
}
|
||||
withhold, redact := d.guard.Rule(info)
|
||||
if withhold {
|
||||
return fmt.Errorf("%s is the file holding the proxy forwarding secret, which is shared cluster-wide, and cannot be downloaded", d.name)
|
||||
}
|
||||
f, err := d.root.Open(d.name)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
d.ContentType = DownloadFileType
|
||||
if !redact {
|
||||
d.file, d.Size = f, info.Size()
|
||||
return nil
|
||||
}
|
||||
defer f.Close()
|
||||
b, err := io.ReadAll(io.LimitReader(f, MaxReadBytes+1))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len(b) > MaxReadBytes {
|
||||
return fmt.Errorf("%s is over %d bytes and cannot be redacted for download", d.name, MaxReadBytes)
|
||||
}
|
||||
d.body = RedactProps(b)
|
||||
d.Size = int64(len(d.body))
|
||||
return nil
|
||||
}
|
||||
|
||||
// Close releases what OpenDownload opened.
|
||||
func (d *Download) Close() error {
|
||||
if d.file != nil {
|
||||
d.file.Close()
|
||||
}
|
||||
return d.root.Close()
|
||||
}
|
||||
|
||||
// WriteTo writes the download to w: exactly Size bytes of a file, or a zip of a
|
||||
// folder whose entries sit under the folder's own name, so unpacking it makes
|
||||
// that one folder. A file that shrank since it was opened is an error, never a
|
||||
// short download passed off as whole.
|
||||
func (d *Download) WriteTo(ctx context.Context, w io.Writer) error {
|
||||
switch {
|
||||
case d.body != nil:
|
||||
_, err := w.Write(d.body)
|
||||
return err
|
||||
case d.file != nil:
|
||||
_, err := io.CopyN(w, ctxReader{ctx, d.file}, d.Size)
|
||||
if errors.Is(err, io.EOF) {
|
||||
return fmt.Errorf("%s shrank while it was being downloaded", d.name)
|
||||
}
|
||||
return err
|
||||
}
|
||||
return d.writeZip(ctx, w)
|
||||
}
|
||||
|
||||
// writeZip streams the folder as a zip. Everything is deflated at the fastest
|
||||
// level: the Job has one CPU and the owner's connection is the slower end, and
|
||||
// already-compressed files (jars, region files) come out as stored blocks
|
||||
// without costing much. Links, devices and sockets are left out, like a world
|
||||
// export leaves them out.
|
||||
func (d *Download) writeZip(ctx context.Context, w io.Writer) error {
|
||||
zw := zip.NewWriter(w)
|
||||
zw.RegisterCompressor(zip.Deflate, func(out io.Writer) (io.WriteCloser, error) {
|
||||
return flate.NewWriter(out, flate.BestSpeed)
|
||||
})
|
||||
base := path.Base(d.name)
|
||||
err := fs.WalkDir(d.root.FS(), d.name, func(p string, de fs.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := ctx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
entry := base + p[len(d.name):]
|
||||
info, err := de.Info()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
switch {
|
||||
case de.IsDir():
|
||||
hdr := &zip.FileHeader{Name: entry + "/", Modified: info.ModTime()}
|
||||
hdr.SetMode(info.Mode().Perm() | fs.ModeDir)
|
||||
_, err := zw.CreateHeader(hdr)
|
||||
return err
|
||||
case !de.Type().IsRegular():
|
||||
d.Skipped++
|
||||
return nil
|
||||
}
|
||||
withhold, redact := d.guard.Rule(info)
|
||||
if withhold {
|
||||
d.Withheld++
|
||||
return nil
|
||||
}
|
||||
f, err := d.root.Open(p)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer f.Close()
|
||||
var src io.Reader = ctxReader{ctx, f}
|
||||
if redact {
|
||||
b, err := io.ReadAll(io.LimitReader(f, MaxReadBytes+1))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len(b) > MaxReadBytes {
|
||||
d.Withheld++
|
||||
return nil
|
||||
}
|
||||
src = bytes.NewReader(RedactProps(b))
|
||||
}
|
||||
hdr := &zip.FileHeader{Name: entry, Method: zip.Deflate, Modified: info.ModTime()}
|
||||
hdr.SetMode(info.Mode().Perm())
|
||||
fw, err := zw.CreateHeader(hdr)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = io.Copy(fw, src)
|
||||
return err
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return zw.Close()
|
||||
}
|
||||
|
||||
// ctxReader stops a long copy once ctx is done.
|
||||
type ctxReader struct {
|
||||
ctx context.Context
|
||||
r io.Reader
|
||||
}
|
||||
|
||||
func (c ctxReader) Read(p []byte) (int, error) {
|
||||
if err := c.ctx.Err(); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return c.r.Read(p)
|
||||
}
|
||||
@@ -0,0 +1,304 @@
|
||||
package fileedit
|
||||
|
||||
import (
|
||||
"archive/zip"
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"syscall"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// downloadWorld is worldRoot plus the guarded files with real secrets in them.
|
||||
func downloadWorld(t *testing.T) string {
|
||||
t.Helper()
|
||||
root, _ := worldRoot(t)
|
||||
for name, body := range map[string]string{
|
||||
"server.properties": "motd=hi\nrcon.password=hunter2\n",
|
||||
"config/paper-global.yml": "secret: aVeryRealForwardingKey\n",
|
||||
} {
|
||||
if err := os.WriteFile(filepath.Join(root, name), []byte(body), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
return root
|
||||
}
|
||||
|
||||
// download runs a whole download into memory.
|
||||
func download(t *testing.T, root, name string, dir bool) (*Download, []byte) {
|
||||
t.Helper()
|
||||
d, err := OpenDownload(root, name, dir)
|
||||
if err != nil {
|
||||
t.Fatalf("OpenDownload(%s): %v", name, err)
|
||||
}
|
||||
defer d.Close()
|
||||
var out bytes.Buffer
|
||||
if err := d.WriteTo(context.Background(), &out); err != nil {
|
||||
t.Fatalf("WriteTo(%s): %v", name, err)
|
||||
}
|
||||
return d, out.Bytes()
|
||||
}
|
||||
|
||||
// unzipped reads a zip into name → content ("<dir> <mode>" for folders).
|
||||
func unzipped(t *testing.T, b []byte) map[string]string {
|
||||
t.Helper()
|
||||
zr, err := zip.NewReader(bytes.NewReader(b), int64(len(b)))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got := map[string]string{}
|
||||
for _, f := range zr.File {
|
||||
if f.FileInfo().IsDir() {
|
||||
// The Unix mode, S_IFDIR included, which is what unzip tools
|
||||
// restore a folder's permissions from.
|
||||
got[f.Name] = fmt.Sprintf("<dir %o>", f.ExternalAttrs>>16)
|
||||
continue
|
||||
}
|
||||
rc, err := f.Open()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
body, err := io.ReadAll(rc)
|
||||
rc.Close()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got[f.Name] = f.Mode().Perm().String() + " " + string(body)
|
||||
}
|
||||
return got
|
||||
}
|
||||
|
||||
func TestDownload(t *testing.T) {
|
||||
const redacted = "motd=hi\nrcon.password=" + redactedValue + "\n"
|
||||
|
||||
t.Run("a file goes out as its exact bytes", func(t *testing.T) {
|
||||
root := downloadWorld(t)
|
||||
d, got := download(t, root, "config/paper.yml", false)
|
||||
if string(got) != "verbose: false\n" || d.Size != int64(len(got)) || d.ContentType != DownloadFileType {
|
||||
t.Fatalf("download = %q, size %d, type %s", got, d.Size, d.ContentType)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("server.properties, under any name, goes out redacted", func(t *testing.T) {
|
||||
root := downloadWorld(t)
|
||||
if err := os.Link(filepath.Join(root, "server.properties"), filepath.Join(root, "copy.txt")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
symlink(t, "server.properties", filepath.Join(root, "sym.txt"))
|
||||
for _, name := range []string{"server.properties", "./server.properties", "copy.txt", "sym.txt"} {
|
||||
d, got := download(t, root, name, false)
|
||||
if string(got) != redacted || d.Size != int64(len(redacted)) {
|
||||
t.Errorf("%s: download = %q, size %d; want %q", name, got, d.Size, redacted)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("the forwarding secret, under any name, is refused", func(t *testing.T) {
|
||||
root := downloadWorld(t)
|
||||
if err := os.Link(filepath.Join(root, "config/paper-global.yml"), filepath.Join(root, "hard.yml")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
symlink(t, "config", filepath.Join(root, "cfg"))
|
||||
for _, name := range []string{"config/paper-global.yml", "hard.yml", "cfg/paper-global.yml"} {
|
||||
d, err := OpenDownload(root, name, false)
|
||||
if err == nil {
|
||||
d.Close()
|
||||
t.Errorf("%s: opened for download", name)
|
||||
} else if !strings.Contains(err.Error(), "forwarding secret") {
|
||||
t.Errorf("%s: err = %v", name, err)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
// A guarded name that is itself a link guards what it points at: that file
|
||||
// is what the server reads, under whatever name it is reached.
|
||||
t.Run("a guarded name that is a link guards its target", func(t *testing.T) {
|
||||
root, _ := worldRoot(t)
|
||||
for name, body := range map[string]string{
|
||||
"config/real.yml": "secret: aVeryRealForwardingKey\n",
|
||||
"real.properties": "motd=hi\nrcon.password=hunter2\n",
|
||||
} {
|
||||
if err := os.WriteFile(filepath.Join(root, name), []byte(body), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if err := os.Remove(filepath.Join(root, "server.properties")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
symlink(t, "real.yml", filepath.Join(root, "config/paper-global.yml"))
|
||||
symlink(t, "real.properties", filepath.Join(root, "server.properties"))
|
||||
if d, err := OpenDownload(root, "config/real.yml", false); err == nil {
|
||||
d.Close()
|
||||
t.Error("the forwarding secret opened under its link target's name")
|
||||
}
|
||||
if _, got := download(t, root, "real.properties", false); string(got) != redacted {
|
||||
t.Errorf("real.properties = %q, want %q", got, redacted)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("what is not there as the listing said is refused", func(t *testing.T) {
|
||||
root := downloadWorld(t)
|
||||
// Opening a FIFO for reading would wait for a writer that never comes.
|
||||
if err := syscall.Mkfifo(filepath.Join(root, "pipe"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, c := range []struct {
|
||||
name string
|
||||
dir bool
|
||||
want string
|
||||
}{
|
||||
{"config", false, "is a folder now"},
|
||||
{"server.properties", true, "is not a folder now"},
|
||||
{"missing.txt", false, "no such file"},
|
||||
{"pipe", false, "not a regular file"},
|
||||
{".", true, "not a file or folder inside"},
|
||||
{"", true, "not a file or folder inside"},
|
||||
{"../outside", true, "not a file or folder inside"},
|
||||
{"/etc", true, "not a file or folder inside"},
|
||||
} {
|
||||
d, err := OpenDownload(root, c.name, c.dir)
|
||||
if err == nil {
|
||||
d.Close()
|
||||
t.Errorf("%q: opened", c.name)
|
||||
} else if !strings.Contains(err.Error(), c.want) {
|
||||
t.Errorf("%q: err = %v, want %q", c.name, err, c.want)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("a folder goes out as a zip under its own name, guarded", func(t *testing.T) {
|
||||
root := downloadWorld(t)
|
||||
plugins := filepath.Join(root, "plugins")
|
||||
for _, d := range []string{"plugins/Essentials/empty", "plugins/Essentials/data"} {
|
||||
if err := os.MkdirAll(filepath.Join(root, d), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
for name, body := range map[string]string{
|
||||
"plugins/a.jar": "jar",
|
||||
"plugins/Essentials/config.yml": "x: 1",
|
||||
"plugins/Essentials/data/server.properties": "rcon.password=notthereal\n",
|
||||
} {
|
||||
if err := os.WriteFile(filepath.Join(root, name), []byte(body), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
for name, mode := range map[string]os.FileMode{"a.jar": 0o755, "Essentials/empty": 0o700} {
|
||||
if err := os.Chmod(filepath.Join(plugins, name), mode); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if err := os.Link(filepath.Join(root, "config/paper-global.yml"), filepath.Join(plugins, "stolen.yml")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.Link(filepath.Join(root, "server.properties"), filepath.Join(plugins, "props.txt")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
symlink(t, "../config/paper-global.yml", filepath.Join(plugins, "sym.yml"))
|
||||
|
||||
d, b := download(t, root, "plugins", true)
|
||||
if d.Size != -1 || d.ContentType != DownloadZipType {
|
||||
t.Fatalf("size %d, type %s", d.Size, d.ContentType)
|
||||
}
|
||||
want := map[string]string{
|
||||
"plugins/": "<dir 40755>",
|
||||
"plugins/Essentials/": "<dir 40755>",
|
||||
"plugins/Essentials/empty/": "<dir 40700>",
|
||||
"plugins/Essentials/data/": "<dir 40755>",
|
||||
"plugins/a.jar": "-rwxr-xr-x jar",
|
||||
"plugins/Essentials/config.yml": "-rw-r--r-- x: 1",
|
||||
// Only the world root's server.properties is the server's.
|
||||
"plugins/Essentials/data/server.properties": "-rw-r--r-- rcon.password=notthereal\n",
|
||||
"plugins/props.txt": "-rw-r--r-- " + redacted,
|
||||
}
|
||||
if got := unzipped(t, b); !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("zip = %v\nwant %v", got, want)
|
||||
}
|
||||
if d.Skipped != 1 || d.Withheld != 1 {
|
||||
t.Errorf("skipped %d, withheld %d; want 1 and 1", d.Skipped, d.Withheld)
|
||||
}
|
||||
if bytes.Contains(b, []byte("aVeryReal")) || bytes.Contains(b, []byte("hunter2")) {
|
||||
t.Fatal("a secret is in the zip")
|
||||
}
|
||||
|
||||
// A nested folder unpacks as itself, not under its parents.
|
||||
_, b = download(t, root, "plugins/Essentials/data", true)
|
||||
want = map[string]string{
|
||||
"data/": "<dir 40755>",
|
||||
"data/server.properties": "-rw-r--r-- rcon.password=notthereal\n",
|
||||
}
|
||||
if got := unzipped(t, b); !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("nested zip = %v\nwant %v", got, want)
|
||||
}
|
||||
})
|
||||
|
||||
// Redaction reads the file whole; one too big for that is refused, or left
|
||||
// out of a folder, never sent as it is.
|
||||
t.Run("a server.properties too big to redact is refused", func(t *testing.T) {
|
||||
root := downloadWorld(t)
|
||||
big := append([]byte("rcon.password=hunter2\n"), bytes.Repeat([]byte("#"), MaxReadBytes)...)
|
||||
if err := os.WriteFile(filepath.Join(root, "server.properties"), big, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
d, err := OpenDownload(root, "server.properties", false)
|
||||
if err == nil {
|
||||
d.Close()
|
||||
t.Fatal("an oversized server.properties opened for download")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "cannot be redacted") {
|
||||
t.Fatalf("err = %v", err)
|
||||
}
|
||||
|
||||
if err := os.Link(filepath.Join(root, "server.properties"), filepath.Join(root, "config/props.txt")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
d, b := download(t, root, "config", true)
|
||||
want := map[string]string{"config/": "<dir 40755>", "config/paper.yml": "-rw-r--r-- verbose: false\n"}
|
||||
if got := unzipped(t, b); !reflect.DeepEqual(got, want) || d.Withheld != 2 {
|
||||
t.Fatalf("zip = %v, withheld %d; want %v and 2", got, d.Withheld, want)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("a file that shrinks mid-download fails it", func(t *testing.T) {
|
||||
root := downloadWorld(t)
|
||||
d, err := OpenDownload(root, "config/paper.yml", false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer d.Close()
|
||||
if err := os.Truncate(filepath.Join(root, "config/paper.yml"), 3); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := d.WriteTo(context.Background(), io.Discard); err == nil || !strings.Contains(err.Error(), "shrank") {
|
||||
t.Fatalf("err = %v, want the shrank error", err)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("a cancelled download stops", func(t *testing.T) {
|
||||
root := downloadWorld(t)
|
||||
// Folders only: no file copy is there to notice the cancel.
|
||||
if err := os.MkdirAll(filepath.Join(root, "empty/a/b"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
for _, c := range []struct {
|
||||
name string
|
||||
dir bool
|
||||
}{{"config/paper.yml", false}, {"config", true}, {"empty", true}} {
|
||||
d, err := OpenDownload(root, c.name, c.dir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := d.WriteTo(ctx, io.Discard); err != context.Canceled {
|
||||
t.Errorf("%s: err = %v, want context.Canceled", c.name, err)
|
||||
}
|
||||
d.Close()
|
||||
}
|
||||
})
|
||||
}
|
||||
+134
-25
@@ -79,13 +79,12 @@ var (
|
||||
ErrExists = errors.New("fileedit: the target already exists")
|
||||
)
|
||||
|
||||
// Runner is the cluster-side half of one file operation: render and create the
|
||||
// Job, wait for its Pod to reach a terminal phase, and return the marked JSON
|
||||
// payload the Pod printed. It is one method rather than a create/poll/read trio
|
||||
// because felis-api cannot poll a Job at all (no jobs:get — see FilesJobName), so
|
||||
// there is no intermediate state a caller could usefully observe; the operation is
|
||||
// synchronous from the API's point of view whether or not the seam pretends
|
||||
// otherwise.
|
||||
// Runner is the cluster-side half of a file operation. Run renders and creates
|
||||
// the Job, waits for its Pod to reach a terminal phase, and returns the marked
|
||||
// JSON payload the Pod printed: from the API's point of view an operation is one
|
||||
// call. An upload too big for one request and an unzip can outlast any request,
|
||||
// so those two are started instead (Start) and read back later (Ops), from the
|
||||
// Jobs felis-api lists and the progress lines their Pods print.
|
||||
//
|
||||
// It is an interface so the Editor's orchestration and error mapping are tested
|
||||
// against a fake; the client-go implementation (K8sRunner) is integration-only.
|
||||
@@ -93,6 +92,11 @@ type Runner interface {
|
||||
// Run creates the Job for p and returns the raw JSON payload from the
|
||||
// ResultPrefix line of its Pod's log.
|
||||
Run(ctx context.Context, p JobParams) ([]byte, error)
|
||||
// Start creates the Job for p and returns once it exists.
|
||||
Start(ctx context.Context, p JobParams) error
|
||||
// Ops reports the background operations (JobParams.Async) of one server
|
||||
// whose Jobs the cluster still holds, newest first.
|
||||
Ops(ctx context.Context, namespace, server string) ([]OpState, error)
|
||||
}
|
||||
|
||||
// Config parameterises the file editor. Image has no default on purpose: it is
|
||||
@@ -139,6 +143,15 @@ type Config struct {
|
||||
// it must stay comfortably longer than the moment felis-api needs to read the
|
||||
// Pod's log, because the TTL takes the Pod (and its log) with the Job.
|
||||
TTLAfterFinished time.Duration
|
||||
|
||||
// AsyncDeadline, AsyncTTL and AsyncCPULimit stand in for Deadline,
|
||||
// TTLAfterFinished and CPULimit on an upload or unzip felis-api starts and
|
||||
// does not wait on. Such a Job moves a whole archive or a file of gigabytes,
|
||||
// so it gets hours; it stays after finishing long enough for the panel to
|
||||
// show how it ended; and it gets a whole core, since inflating is CPU-bound.
|
||||
AsyncDeadline time.Duration
|
||||
AsyncTTL time.Duration
|
||||
AsyncCPULimit string
|
||||
}
|
||||
|
||||
// defaults applied when a Config field is left zero. They are sized for what a
|
||||
@@ -153,6 +166,9 @@ const (
|
||||
defaultCPULimit = "500m"
|
||||
defaultMemLimit = "256Mi"
|
||||
defaultTTL = 2 * time.Minute
|
||||
defaultAsyncDeadline = 2 * time.Hour
|
||||
defaultAsyncTTL = 30 * time.Minute
|
||||
defaultAsyncCPULimit = "1"
|
||||
)
|
||||
|
||||
// withDefaults returns a copy of c with zero fields filled, so a partially
|
||||
@@ -182,6 +198,15 @@ func (c Config) withDefaults() Config {
|
||||
if c.TTLAfterFinished <= 0 {
|
||||
c.TTLAfterFinished = defaultTTL
|
||||
}
|
||||
if c.AsyncDeadline <= 0 {
|
||||
c.AsyncDeadline = defaultAsyncDeadline
|
||||
}
|
||||
if c.AsyncTTL <= 0 {
|
||||
c.AsyncTTL = defaultAsyncTTL
|
||||
}
|
||||
if c.AsyncCPULimit == "" {
|
||||
c.AsyncCPULimit = defaultAsyncCPULimit
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
@@ -191,19 +216,29 @@ type Editor struct {
|
||||
Config Config
|
||||
}
|
||||
|
||||
// Listing is one directory as List returns it.
|
||||
type Listing struct {
|
||||
Entries []Entry
|
||||
// Truncated reports that the directory holds more than MaxEntries.
|
||||
Truncated bool
|
||||
// Free is the bytes free on the server's volume, 0 when the Job could not
|
||||
// tell.
|
||||
Free int64
|
||||
}
|
||||
|
||||
// List returns one directory's entries, resolved under the server's world root.
|
||||
// An empty path lists the world root itself.
|
||||
func (e *Editor) List(ctx context.Context, server, path string) ([]Entry, bool, error) {
|
||||
func (e *Editor) List(ctx context.Context, server, path string) (Listing, error) {
|
||||
res, err := e.run(ctx, server, JobParams{Op: OpList, Path: path})
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
return Listing{}, err
|
||||
}
|
||||
// A genuinely empty directory unmarshals Entries as nil; normalise it so the
|
||||
// handler serialises [] rather than null.
|
||||
if res.Entries == nil {
|
||||
res.Entries = []Entry{}
|
||||
}
|
||||
return res.Entries, res.Truncated, nil
|
||||
return Listing{Entries: res.Entries, Truncated: res.Truncated, Free: res.Avail}, nil
|
||||
}
|
||||
|
||||
// Read returns a file's bytes, resolved under the server's world root, and the
|
||||
@@ -289,7 +324,7 @@ func (e *Editor) run(ctx context.Context, server string, p JobParams) (Result, e
|
||||
}
|
||||
|
||||
cfg := e.Config.withDefaults()
|
||||
opID, err := newOpID()
|
||||
p, err := cfg.params(server, p)
|
||||
if err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
@@ -301,20 +336,6 @@ func (e *Editor) run(ctx context.Context, server string, p JobParams) (Result, e
|
||||
ctx, cancel := context.WithTimeout(ctx, cfg.Timeout)
|
||||
defer cancel()
|
||||
|
||||
p.Server = server
|
||||
p.OpID = opID
|
||||
p.WorldPVC = naming.WorldPVCName(server)
|
||||
p.Namespace = cfg.Namespace
|
||||
p.ServiceAccount = cfg.ServiceAccount
|
||||
p.Image = cfg.Image
|
||||
p.WorldsRoot = cfg.WorldsRoot
|
||||
p.Deadline = cfg.Deadline
|
||||
p.CPULimit = cfg.CPULimit
|
||||
p.MemLimit = cfg.MemLimit
|
||||
p.RunAsUser = cfg.RunAsUser
|
||||
p.RunAsGroup = cfg.RunAsGroup
|
||||
p.FSGroup = cfg.FSGroup
|
||||
p.TTLAfterFinished = cfg.TTLAfterFinished
|
||||
payload, err := e.Runner.Run(ctx, p)
|
||||
if err != nil {
|
||||
return Result{}, err
|
||||
@@ -327,6 +348,94 @@ func (e *Editor) run(ctx context.Context, server string, p JobParams) (Result, e
|
||||
return res, resultError(res)
|
||||
}
|
||||
|
||||
// params fills in what every Job of server takes from the Config, and a fresh op
|
||||
// id; p carries the op and its own fields.
|
||||
func (c Config) params(server string, p JobParams) (JobParams, error) {
|
||||
opID, err := newOpID()
|
||||
if err != nil {
|
||||
return JobParams{}, err
|
||||
}
|
||||
p.Server = server
|
||||
p.OpID = opID
|
||||
p.WorldPVC = naming.WorldPVCName(server)
|
||||
p.Namespace = c.Namespace
|
||||
p.ServiceAccount = c.ServiceAccount
|
||||
p.Image = c.Image
|
||||
p.WorldsRoot = c.WorldsRoot
|
||||
p.Deadline = c.Deadline
|
||||
p.CPULimit = c.CPULimit
|
||||
p.MemLimit = c.MemLimit
|
||||
p.RunAsUser = c.RunAsUser
|
||||
p.RunAsGroup = c.RunAsGroup
|
||||
p.FSGroup = c.FSGroup
|
||||
p.TTLAfterFinished = c.TTLAfterFinished
|
||||
return p, nil
|
||||
}
|
||||
|
||||
// The states of an OpState.
|
||||
const (
|
||||
OpRunning = "running"
|
||||
OpSucceeded = "succeeded"
|
||||
OpFailed = "failed"
|
||||
)
|
||||
|
||||
// OpState is where one background file operation stands.
|
||||
type OpState struct {
|
||||
ID string
|
||||
Op string
|
||||
Path string
|
||||
State string
|
||||
// Started is when the Job was created; Finished when it ended, zero while it
|
||||
// runs.
|
||||
Started time.Time
|
||||
Finished time.Time
|
||||
// Done and Total are the bytes of the latest progress line, zero before the
|
||||
// first.
|
||||
Done, Total int64
|
||||
// Result is what the Job printed once it finished. It is nil while the Job
|
||||
// runs, and for a Job that ended without printing one (killed at its
|
||||
// deadline, out of memory, its bytes unfetchable), whose Reason says why.
|
||||
Result *Result
|
||||
Reason string
|
||||
}
|
||||
|
||||
// StartUpload starts landing the staged bytes src describes at path and returns
|
||||
// without waiting, for a file too big to land inside one request (Upload). The
|
||||
// Job checks what Upload's does; Ops reports how it ends.
|
||||
func (e *Editor) StartUpload(ctx context.Context, server, path string, src UploadSource, overwrite bool) (OpState, error) {
|
||||
return e.start(ctx, server, JobParams{
|
||||
Op: OpUpload, Path: path, Overwrite: overwrite,
|
||||
SourceURL: src.URL, UploadToken: src.Token, UploadSize: src.Size, UploadSHA256: src.SHA256,
|
||||
})
|
||||
}
|
||||
|
||||
// StartUnzip starts extracting the .zip at path into the folder holding it and
|
||||
// returns without waiting. Without overwrite an archive that would replace a
|
||||
// file changes nothing and ends with CodeExists and the list (Result.Conflicts).
|
||||
func (e *Editor) StartUnzip(ctx context.Context, server, path string, overwrite bool) (OpState, error) {
|
||||
return e.start(ctx, server, JobParams{Op: OpUnzip, Path: path, Overwrite: overwrite})
|
||||
}
|
||||
|
||||
// Ops reports the server's background operations the cluster still holds: the
|
||||
// one running, if any, and those finished within AsyncTTL.
|
||||
func (e *Editor) Ops(ctx context.Context, server string) ([]OpState, error) {
|
||||
return e.Runner.Ops(ctx, e.Config.withDefaults().Namespace, server)
|
||||
}
|
||||
|
||||
func (e *Editor) start(ctx context.Context, server string, p JobParams) (OpState, error) {
|
||||
cfg := e.Config.withDefaults()
|
||||
p, err := cfg.params(server, p)
|
||||
if err != nil {
|
||||
return OpState{}, err
|
||||
}
|
||||
p.Async = true
|
||||
p.Deadline, p.TTLAfterFinished, p.CPULimit = cfg.AsyncDeadline, cfg.AsyncTTL, cfg.AsyncCPULimit
|
||||
if err := e.Runner.Start(ctx, p); err != nil {
|
||||
return OpState{}, err
|
||||
}
|
||||
return OpState{ID: p.OpID, Op: p.Op, Path: p.Path, State: OpRunning, Started: time.Now()}, nil
|
||||
}
|
||||
|
||||
// resultError translates a Result's code into the sentinel the API maps. An
|
||||
// unrecognised code is deliberately NOT swallowed as success: a Job reporting a
|
||||
// failure this build does not know about must still fail the request, or a future
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// fakeRunner stands in for the cluster: it records the JobParams the Editor
|
||||
@@ -14,6 +15,21 @@ type fakeRunner struct {
|
||||
got []JobParams
|
||||
payload []byte
|
||||
err error
|
||||
|
||||
started []JobParams
|
||||
startErr error
|
||||
ops []OpState
|
||||
opsArgs [][2]string
|
||||
}
|
||||
|
||||
func (f *fakeRunner) Start(_ context.Context, p JobParams) error {
|
||||
f.started = append(f.started, p)
|
||||
return f.startErr
|
||||
}
|
||||
|
||||
func (f *fakeRunner) Ops(_ context.Context, namespace, server string) ([]OpState, error) {
|
||||
f.opsArgs = append(f.opsArgs, [2]string{namespace, server})
|
||||
return f.ops, f.err
|
||||
}
|
||||
|
||||
func (f *fakeRunner) Run(_ context.Context, p JobParams) ([]byte, error) {
|
||||
@@ -37,15 +53,15 @@ func mustPayload(t *testing.T, res Result) []byte {
|
||||
// the editor pointed at the same volume the operator created and the reaper deletes.
|
||||
func TestEditorRendersParams(t *testing.T) {
|
||||
t.Run("list", func(t *testing.T) {
|
||||
r := &fakeRunner{payload: mustPayload(t, Result{Entries: []Entry{{Name: "a"}}})}
|
||||
r := &fakeRunner{payload: mustPayload(t, Result{Entries: []Entry{{Name: "a"}}, Avail: 7 << 30})}
|
||||
e := &Editor{Runner: r, Config: Config{Image: "img"}}
|
||||
|
||||
entries, truncated, err := e.List(context.Background(), "survival", "config")
|
||||
ls, err := e.List(context.Background(), "survival", "config")
|
||||
if err != nil {
|
||||
t.Fatalf("List: %v", err)
|
||||
}
|
||||
if len(entries) != 1 || truncated {
|
||||
t.Fatalf("entries=%+v truncated=%v", entries, truncated)
|
||||
if len(ls.Entries) != 1 || ls.Truncated || ls.Free != 7<<30 {
|
||||
t.Fatalf("listing = %+v", ls)
|
||||
}
|
||||
p := r.got[0]
|
||||
if p.Op != OpList || p.Path != "config" || p.Server != "survival" {
|
||||
@@ -242,11 +258,11 @@ func TestEditorNormalisesEmptyResults(t *testing.T) {
|
||||
r := &fakeRunner{payload: mustPayload(t, Result{})}
|
||||
e := &Editor{Runner: r, Config: Config{Image: "img"}}
|
||||
|
||||
entries, _, err := e.List(context.Background(), "survival", "empty")
|
||||
ls, err := e.List(context.Background(), "survival", "empty")
|
||||
if err != nil {
|
||||
t.Fatalf("List: %v", err)
|
||||
}
|
||||
if entries == nil {
|
||||
if ls.Entries == nil {
|
||||
t.Fatal("an empty directory must list as [], not nil")
|
||||
}
|
||||
|
||||
@@ -297,3 +313,85 @@ func TestExtractResult(t *testing.T) {
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// TestEditorStartsBackgroundOps checks an upload or unzip too long to wait on is
|
||||
// started, not run: its Job carries the async label and the longer deadline,
|
||||
// the longer TTL Ops reads it back within, and the larger CPU share, and what
|
||||
// comes back names the Job Ops will report on.
|
||||
func TestEditorStartsBackgroundOps(t *testing.T) {
|
||||
src := UploadSource{URL: "http://api/big", Token: "tok", Size: 5 << 30, SHA256: "sum"}
|
||||
|
||||
t.Run("upload", func(t *testing.T) {
|
||||
r := &fakeRunner{}
|
||||
e := &Editor{Runner: r, Config: Config{Image: "img", Namespace: "mc"}}
|
||||
before := time.Now()
|
||||
st, err := e.StartUpload(context.Background(), "survival", "maps/world.zip", src, true)
|
||||
if err != nil {
|
||||
t.Fatalf("StartUpload: %v", err)
|
||||
}
|
||||
if r.calls != 0 || len(r.started) != 1 {
|
||||
t.Fatalf("ran %d, started %d; want the one Job started and none waited on", r.calls, len(r.started))
|
||||
}
|
||||
p := r.started[0]
|
||||
if !p.Async || p.Op != OpUpload || p.Path != "maps/world.zip" || !p.Overwrite ||
|
||||
p.SourceURL != src.URL || p.UploadToken != "tok" || p.UploadSize != 5<<30 || p.UploadSHA256 != "sum" {
|
||||
t.Fatalf("params = %+v", p)
|
||||
}
|
||||
if p.Deadline != 2*time.Hour || p.TTLAfterFinished != 30*time.Minute || p.CPULimit != "1" || p.MemLimit != "256Mi" {
|
||||
t.Fatalf("deadline %v ttl %v cpu %q mem %q, want 2h 30m 1 256Mi",
|
||||
p.Deadline, p.TTLAfterFinished, p.CPULimit, p.MemLimit)
|
||||
}
|
||||
if p.Namespace != "mc" || p.Server != "survival" || p.WorldPVC != "world-survival-0" || p.Image != "img" || p.OpID == "" {
|
||||
t.Fatalf("params = %+v", p)
|
||||
}
|
||||
if st.ID != p.OpID || st.Op != OpUpload || st.Path != "maps/world.zip" || st.State != OpRunning || st.Started.Before(before) {
|
||||
t.Fatalf("state = %+v, want the started Job %s running", st, p.OpID)
|
||||
}
|
||||
|
||||
if _, err := e.StartUpload(context.Background(), "survival", "maps/world.zip", src, false); err != nil || r.started[1].Overwrite {
|
||||
t.Fatalf("an upload that must not replace a file started with %+v (%v)", r.started[1], err)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("unzip, with its own limits", func(t *testing.T) {
|
||||
r := &fakeRunner{}
|
||||
e := &Editor{Runner: r, Config: Config{Image: "img", AsyncDeadline: time.Hour, AsyncTTL: time.Minute, AsyncCPULimit: "2"}}
|
||||
st, err := e.StartUnzip(context.Background(), "survival", "maps/world.zip", false)
|
||||
if err != nil {
|
||||
t.Fatalf("StartUnzip: %v", err)
|
||||
}
|
||||
p := r.started[0]
|
||||
if !p.Async || p.Op != OpUnzip || p.Path != "maps/world.zip" || p.Overwrite || p.SourceURL != "" {
|
||||
t.Fatalf("params = %+v", p)
|
||||
}
|
||||
if p.Deadline != time.Hour || p.TTLAfterFinished != time.Minute || p.CPULimit != "2" {
|
||||
t.Fatalf("deadline %v ttl %v cpu %q, want the configured 1h 1m 2", p.Deadline, p.TTLAfterFinished, p.CPULimit)
|
||||
}
|
||||
if st.ID != p.OpID || st.Op != OpUnzip {
|
||||
t.Fatalf("state = %+v", st)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("a Job that could not be created", func(t *testing.T) {
|
||||
boom := errors.New("forbidden")
|
||||
r := &fakeRunner{startErr: boom}
|
||||
e := &Editor{Runner: r, Config: Config{Image: "img"}}
|
||||
st, err := e.StartUnzip(context.Background(), "survival", "a.zip", false)
|
||||
if !errors.Is(err, boom) || st != (OpState{}) {
|
||||
t.Fatalf("state %+v err %v, want nothing started and %v", st, err, boom)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("ops", func(t *testing.T) {
|
||||
want := []OpState{{ID: "0a", State: OpRunning}}
|
||||
r := &fakeRunner{ops: want}
|
||||
e := &Editor{Runner: r, Config: Config{Image: "img"}}
|
||||
got, err := e.Ops(context.Background(), "survival")
|
||||
if err != nil || len(got) != 1 || got[0] != want[0] {
|
||||
t.Fatalf("Ops = %+v %v", got, err)
|
||||
}
|
||||
if r.opsArgs[0] != [2]string{"minecraft", "survival"} {
|
||||
t.Fatalf("asked %v, want the default namespace and the server", r.opsArgs[0])
|
||||
}
|
||||
})
|
||||
}
|
||||
+81
-29
@@ -10,6 +10,7 @@ import (
|
||||
"fmt"
|
||||
"io"
|
||||
"io/fs"
|
||||
"math"
|
||||
"os"
|
||||
"path"
|
||||
"strings"
|
||||
@@ -20,11 +21,12 @@ import (
|
||||
)
|
||||
|
||||
// The operations the editor supports: list a directory, read a file, write a
|
||||
// file, make a directory, delete, rename, and upload. The set is closed; there is
|
||||
// no chmod, chown, link or copy. Every op resolves every path through os.Root (see
|
||||
// Execute), and each mutating op carries its own containment note below.
|
||||
// file, make a directory, delete, rename, upload, and extract a .zip. The set is
|
||||
// closed; there is no chmod, chown, link or copy. Every op resolves every path
|
||||
// through os.Root (see Execute), and each mutating op carries its own
|
||||
// containment note below.
|
||||
//
|
||||
// A write or upload DOES land arbitrary bytes at any path inside the mount, and
|
||||
// A write, upload or unzip DOES land arbitrary bytes at any path inside the mount, and
|
||||
// that is a real capability rather than an oversight: the root is the server's
|
||||
// whole working directory (see Config.WorldsRoot), so an owner can upload
|
||||
// plugins/<x>.jar and Paper will load it on the next boot. It is the same power a
|
||||
@@ -33,8 +35,8 @@ import (
|
||||
// image curation. Images are admin-only (POST /images, POST /images/build) and
|
||||
// modpack submissions need an admin verdict, so this is the one owner-tier route
|
||||
// that lands executable code in a backend pod. That trade was made deliberately;
|
||||
// if it is ever revisited, the guard belongs in land() below, which is the single
|
||||
// choke point both byte-landing ops route through.
|
||||
// if it is ever revisited, the guard belongs in land() below, the choke point
|
||||
// write and upload route through, and in unzip's extractOne (unzip.go).
|
||||
const (
|
||||
OpList = "list"
|
||||
OpRead = "read"
|
||||
@@ -43,6 +45,7 @@ const (
|
||||
OpDelete = "delete"
|
||||
OpRename = "rename"
|
||||
OpUpload = "upload"
|
||||
OpUnzip = "unzip"
|
||||
)
|
||||
|
||||
// mutates reports whether op changes the world, and so whether its Job gets the
|
||||
@@ -54,7 +57,7 @@ func mutates(op string) bool { return op != OpList && op != OpRead }
|
||||
// validOp reports whether op is one the Job knows.
|
||||
func validOp(op string) bool {
|
||||
switch op {
|
||||
case OpList, OpRead, OpWrite, OpMkdir, OpDelete, OpRename, OpUpload:
|
||||
case OpList, OpRead, OpWrite, OpMkdir, OpDelete, OpRename, OpUpload, OpUnzip:
|
||||
return true
|
||||
}
|
||||
return false
|
||||
@@ -133,12 +136,12 @@ const UploadTokenEnv = "FELIS_UPLOAD_TOKEN"
|
||||
// - MaxEntries bounds a listing. A world's region/ directory legitimately holds
|
||||
// thousands of .mca files, so this truncates rather than errors (Truncated
|
||||
// says so), keeping the log line bounded while still being useful.
|
||||
// - MaxUploadBytes bounds an upload. Its bytes travel neither through the Job
|
||||
// spec nor the pod log — felis-api stages them and the Job fetches them — so
|
||||
// the bound is the request body instead: the Cloudflare edge refuses bodies
|
||||
// over 100 MB on the Free and Pro plans, and 64 MiB covers the largest plugin
|
||||
// jars (a Geyser build is about 20 MiB) with room to spare. A whole world is
|
||||
// a different operation (a restore), not an upload.
|
||||
// - MaxUploadBytes bounds an upload sent as ONE request body: the Cloudflare
|
||||
// edge refuses bodies over 100 MB on the Free and Pro plans, and 64 MiB
|
||||
// covers the largest plugin jars (a Geyser build is about 20 MiB) with room
|
||||
// to spare. It is felis-api's bound on that route only. A bigger file arrives
|
||||
// in parts and is bounded by nothing but the room on the server's volume,
|
||||
// which the Job checks before it fetches a byte (upload).
|
||||
const (
|
||||
MaxWriteBytes = 256 << 10 // 256 KiB
|
||||
MaxReadBytes = 1 << 20 // 1 MiB
|
||||
@@ -183,6 +186,22 @@ type Result struct {
|
||||
// conflict, the file as it is now. A client hands it back as the expected
|
||||
// hash of its next write (see write).
|
||||
SHA256 string `json:"sha256,omitempty"`
|
||||
|
||||
// Conflicts lists, relative to the root and sorted, the existing files an
|
||||
// unzip would replace: the first MaxConflicts of them. ConflictCount is how
|
||||
// many there are in all.
|
||||
Conflicts []string `json:"conflicts,omitempty"`
|
||||
ConflictCount int `json:"conflict_count,omitempty"`
|
||||
// Entry names what an unzip refused: the archive entry, or the path on the
|
||||
// server it collides with.
|
||||
Entry string `json:"entry,omitempty"`
|
||||
// Need and Avail are, on a no_space an upload or unzip saw coming, the bytes
|
||||
// it needs and the bytes the volume has free. A listing sets Avail too.
|
||||
Need int64 `json:"need,omitempty"`
|
||||
Avail int64 `json:"avail,omitempty"`
|
||||
// Files and Bytes are what a successful unzip extracted.
|
||||
Files int `json:"files,omitempty"`
|
||||
Bytes int64 `json:"bytes,omitempty"`
|
||||
}
|
||||
|
||||
// Request is one file operation. Op decides which of the other fields it reads.
|
||||
@@ -200,10 +219,15 @@ type Request struct {
|
||||
// panel's "new file", which must never truncate a file it did not know was
|
||||
// there.
|
||||
CreateOnly bool
|
||||
// Upload is where an upload's bytes come from; Overwrite lets it replace a
|
||||
// file already at the path.
|
||||
// Upload is where an upload's bytes come from. Overwrite lets an upload
|
||||
// replace a file already at the path, and an unzip replace the files it
|
||||
// collides with.
|
||||
Upload *Upload
|
||||
Overwrite bool
|
||||
// Progress, when set, hears how far an upload or unzip has got: bytes landed
|
||||
// so far out of the total. It is called from the copy loop, often; the caller
|
||||
// throttles.
|
||||
Progress func(done, total int64)
|
||||
}
|
||||
|
||||
// Upload describes the bytes an upload lands. Size and SHA256 are what felis-api
|
||||
@@ -262,7 +286,7 @@ func Execute(root string, req Request) (Result, error) {
|
||||
|
||||
switch req.Op {
|
||||
case OpList:
|
||||
return list(r, path), nil
|
||||
return list(r, root, path), nil
|
||||
case OpRead:
|
||||
return read(r, path), nil
|
||||
case OpWrite:
|
||||
@@ -277,7 +301,9 @@ func Execute(root string, req Request) (Result, error) {
|
||||
if req.Upload == nil {
|
||||
return Result{}, errors.New("an upload needs a source")
|
||||
}
|
||||
return upload(r, path, *req.Upload, req.Overwrite)
|
||||
return upload(r, root, path, *req.Upload, req.Overwrite, req.Progress)
|
||||
case OpUnzip:
|
||||
return unzip(r, root, path, req.Overwrite, req.Progress), nil
|
||||
default:
|
||||
return Result{}, fmt.Errorf("unknown op %q", req.Op)
|
||||
}
|
||||
@@ -285,8 +311,10 @@ func Execute(root string, req Request) (Result, error) {
|
||||
|
||||
// list reads one directory. It does not recurse: a browser asks for one level at
|
||||
// a time, and recursion would make both the result size and the traversal cost
|
||||
// unbounded in a world directory.
|
||||
func list(r *os.Root, path string) Result {
|
||||
// unbounded in a world directory. It also reports the room left on the volume
|
||||
// (Avail), so the panel can refuse an upload the volume cannot take before
|
||||
// sending a byte of it.
|
||||
func list(r *os.Root, rootPath, path string) Result {
|
||||
f, err := r.Open(path)
|
||||
if err != nil {
|
||||
return failure(err, path)
|
||||
@@ -318,7 +346,11 @@ func list(r *os.Root, path string) Result {
|
||||
}
|
||||
entries = append(entries, e)
|
||||
}
|
||||
return Result{Entries: entries, Truncated: truncated}
|
||||
res := Result{Entries: entries, Truncated: truncated}
|
||||
if avail, _, err := statfs(rootPath); err == nil {
|
||||
res.Avail = int64(min(avail, math.MaxInt64))
|
||||
}
|
||||
return res
|
||||
}
|
||||
|
||||
// secretConfigPath is the one file in a world mount holding PLATFORM secret
|
||||
@@ -370,6 +402,13 @@ func read(r *os.Root, name string) Result {
|
||||
if info.IsDir() {
|
||||
return Result{Code: CodeBadPath, Error: fmt.Sprintf("%s is a directory, not a file", name)}
|
||||
}
|
||||
// The name check above answers the plain path with a clear reason; this one
|
||||
// catches the same file reached through a link (see Guard).
|
||||
withhold, redact := NewGuard(r).Rule(info)
|
||||
if withhold {
|
||||
return Result{Code: CodeBadPath, Error: fmt.Sprintf(
|
||||
"%s is the file holding the proxy forwarding secret, which is shared cluster-wide, and is not readable through the editor", name)}
|
||||
}
|
||||
if info.Size() > MaxReadBytes {
|
||||
return Result{Code: CodeTooLarge, Error: fmt.Sprintf(
|
||||
"%s is %d bytes; the editor reads at most %d", name, info.Size(), MaxReadBytes)}
|
||||
@@ -382,7 +421,11 @@ func read(r *os.Root, name string) Result {
|
||||
if err != nil {
|
||||
return failure(err, name)
|
||||
}
|
||||
return Result{Content: redactSecretProps(name, b), SHA256: digest(b)}
|
||||
content := b
|
||||
if redact {
|
||||
content = RedactProps(b)
|
||||
}
|
||||
return Result{Content: content, SHA256: digest(b)}
|
||||
}
|
||||
|
||||
// propsPath is the server's main config file, and rconPasswordKey the one line in
|
||||
@@ -515,8 +558,8 @@ func (e *transferError) Error() string { return "fetch upload: " + e.err.Error()
|
||||
func (e *transferError) Unwrap() error { return e.err }
|
||||
|
||||
// land atomically puts the bytes fill writes at target, the path landingTarget
|
||||
// returned for name. It is the single choke point both byte-landing ops (write and
|
||||
// upload) route through.
|
||||
// returned for name. Write and upload both land through it; unzip lands a whole
|
||||
// tree at once and has its own path (unzip.go).
|
||||
//
|
||||
// The bytes go to a temporary sibling that is synced and then renamed over the
|
||||
// target, so a full disk, a Job killed at its deadline or a crashed node leaves
|
||||
@@ -596,15 +639,19 @@ func syncDir(r *os.Root, dir string) {
|
||||
// The fetched bytes must match both the size and the SHA-256 felis-api received;
|
||||
// either mismatch is a broken transfer, and the target is left as it was. A
|
||||
// success has landed exactly what felis-api staged, whose digest it already holds.
|
||||
func upload(r *os.Root, name string, u Upload, overwrite bool) (Result, error) {
|
||||
if u.Size > MaxUploadBytes {
|
||||
return Result{Code: CodeTooLarge, Error: fmt.Sprintf(
|
||||
"the upload is %d bytes; the editor uploads at most %d", u.Size, MaxUploadBytes)}, nil
|
||||
}
|
||||
//
|
||||
// There is no size ceiling here. The volume is the bound, and it is checked up
|
||||
// front: the new bytes land beside the file they replace until the rename, so
|
||||
// they need their whole size free whatever is already at the path.
|
||||
func upload(r *os.Root, rootPath, name string, u Upload, overwrite bool, progress func(done, total int64)) (Result, error) {
|
||||
target, mode, res := landingTarget(r, name, overwrite)
|
||||
if res.Code != "" {
|
||||
return res, nil
|
||||
}
|
||||
if avail, _, err := statfs(rootPath); err == nil && uint64(u.Size) > avail {
|
||||
return Result{Code: CodeNoSpace, Need: u.Size, Avail: int64(min(avail, math.MaxInt64)), Error: fmt.Sprintf(
|
||||
"%s is %d bytes and the server's volume has %d free; nothing was changed", name, u.Size, avail)}, nil
|
||||
}
|
||||
return land(r, name, target, mode, func(w io.Writer) error {
|
||||
body, err := u.Open()
|
||||
if err != nil {
|
||||
@@ -612,8 +659,13 @@ func upload(r *os.Root, name string, u Upload, overwrite bool) (Result, error) {
|
||||
}
|
||||
defer body.Close()
|
||||
h := sha256.New()
|
||||
var done int64
|
||||
out := io.MultiWriter(w, h)
|
||||
if progress != nil {
|
||||
out = countingWriter{out, func(n int) { done += int64(n); progress(done, u.Size) }}
|
||||
}
|
||||
// One byte past Size so a source that sends more than it promised is seen.
|
||||
n, err := io.Copy(io.MultiWriter(w, h), sourceReader{io.LimitReader(body, u.Size+1)})
|
||||
n, err := io.Copy(out, sourceReader{io.LimitReader(body, u.Size+1)})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -2,7 +2,11 @@ package fileedit
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"math"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
@@ -176,6 +180,31 @@ func TestExecuteHappyPath(t *testing.T) {
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("a listing reports the room left on the volume", func(t *testing.T) {
|
||||
prev := statfs
|
||||
t.Cleanup(func() { statfs = prev })
|
||||
var asked string
|
||||
statfs = func(dir string) (uint64, uint64, error) { asked = dir; return 12345, 99999, nil }
|
||||
res, err := run(root, OpList, "config", nil, "")
|
||||
if err != nil || res.Code != "" {
|
||||
t.Fatalf("Execute: %v %+v", err, res)
|
||||
}
|
||||
if res.Avail != 12345 || asked != root {
|
||||
t.Fatalf("avail = %d measured at %q, want 12345 at %q", res.Avail, asked, root)
|
||||
}
|
||||
|
||||
statfs = func(string) (uint64, uint64, error) { return math.MaxUint64, math.MaxUint64, nil }
|
||||
if res, _ := run(root, OpList, "config", nil, ""); res.Avail != math.MaxInt64 {
|
||||
t.Fatalf("avail = %d, want it clamped to %d", res.Avail, int64(math.MaxInt64))
|
||||
}
|
||||
|
||||
statfs = func(string) (uint64, uint64, error) { return 1, 1, errors.New("no statfs") }
|
||||
res, err = run(root, OpList, "config", nil, "")
|
||||
if err != nil || res.Code != "" || len(res.Entries) != 1 || res.Avail != 0 {
|
||||
t.Fatalf("a volume that cannot be measured still lists, with no room reported: %v %+v", err, res)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("read a file", func(t *testing.T) {
|
||||
res, err := run(root, OpRead, "server.properties", nil, "")
|
||||
if err != nil {
|
||||
@@ -396,6 +425,11 @@ func TestReadRedactsRconPassword(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("Execute: %v", err)
|
||||
}
|
||||
// The hash is the file on disk, the one a save's expect_sha256 is checked
|
||||
// against, not the redacted copy the editor shows.
|
||||
if sum := sha256.Sum256([]byte(props)); res.SHA256 != hex.EncodeToString(sum[:]) {
|
||||
t.Fatalf("sha256 = %s, want the hash of the file as stored", res.SHA256)
|
||||
}
|
||||
got := string(res.Content)
|
||||
if strings.Contains(got, "hunter2") {
|
||||
t.Fatalf("read returned the RCON password (spec §286):\n%s", got)
|
||||
@@ -420,6 +454,51 @@ func TestReadRedactsRconPassword(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestReadGuardsLinksToGuardedFiles: a plugin runs as the game uid and can leave
|
||||
// a link to either guarded file anywhere in the world. Read under the link's
|
||||
// name, the forwarding secret is still refused and the RCON password still
|
||||
// redacted, whether the link is symbolic, a hard link, or a linked folder.
|
||||
func TestReadGuardsLinksToGuardedFiles(t *testing.T) {
|
||||
root, _ := worldRoot(t)
|
||||
const secret = "secret: aVeryRealForwardingKey"
|
||||
if err := os.WriteFile(filepath.Join(root, "config", "paper-global.yml"), []byte(secret), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(root, "server.properties"), []byte("motd=hi\nrcon.password=hunter2\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
symlink(t, "config/paper-global.yml", filepath.Join(root, "sym.yml"))
|
||||
symlink(t, "config", filepath.Join(root, "cfg-link"))
|
||||
symlink(t, "server.properties", filepath.Join(root, "sym.properties"))
|
||||
for _, l := range [][2]string{
|
||||
{"config/paper-global.yml", "hard.yml"},
|
||||
{"server.properties", "hard.properties"},
|
||||
} {
|
||||
if err := os.Link(filepath.Join(root, l[0]), filepath.Join(root, l[1])); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
for _, name := range []string{"sym.yml", "cfg-link/paper-global.yml", "hard.yml"} {
|
||||
res, err := run(root, OpRead, name, nil, "")
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", name, err)
|
||||
}
|
||||
if res.Code != CodeBadPath || strings.Contains(string(res.Content), "aVeryReal") {
|
||||
t.Errorf("%s: result = %+v, want bad_path and no secret", name, res)
|
||||
}
|
||||
}
|
||||
for _, name := range []string{"sym.properties", "hard.properties"} {
|
||||
res, err := run(root, OpRead, name, nil, "")
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", name, err)
|
||||
}
|
||||
if want := "motd=hi\nrcon.password=" + redactedValue + "\n"; res.Code != "" || string(res.Content) != want {
|
||||
t.Errorf("%s: result = %+v, want content %q", name, res, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestWriteIsAtomic is the durability contract: a write that fails part-way leaves
|
||||
// the original file byte-for-byte intact and no stray sibling behind, and a write
|
||||
// that succeeds keeps the file's mode.
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
package fileedit
|
||||
|
||||
import (
|
||||
"io/fs"
|
||||
"os"
|
||||
"path"
|
||||
)
|
||||
|
||||
// What leaves a world mount — a read, a download, a world export, a backup
|
||||
// export — passes the same two guards: the forwarding-secret file
|
||||
// (secretConfigPath) is withheld, and server.properties has its RCON password
|
||||
// redacted (propsPath).
|
||||
//
|
||||
// On a live mount both are matched by the file itself (os.SameFile), not by the
|
||||
// name it was reached under. A plugin runs arbitrary code as the game uid and can
|
||||
// leave a symbolic or hard link to either file anywhere in the world; a name
|
||||
// check alone would hand the secret out under the link's name. A stored archive
|
||||
// has only names, so ArchiveRule matches those.
|
||||
|
||||
// Guard knows the two guarded files of one world mount.
|
||||
type Guard struct {
|
||||
secret, props fs.FileInfo
|
||||
}
|
||||
|
||||
// NewGuard looks the guarded files up in r. One that is missing guards nothing:
|
||||
// no file can be the same file as it.
|
||||
func NewGuard(r *os.Root) Guard {
|
||||
var g Guard
|
||||
if fi, err := r.Stat(secretConfigPath); err == nil {
|
||||
g.secret = fi
|
||||
}
|
||||
if fi, err := r.Stat(propsPath); err == nil {
|
||||
g.props = fi
|
||||
}
|
||||
return g
|
||||
}
|
||||
|
||||
// Rule reports whether the file fi describes must be withheld, or sent only
|
||||
// through RedactProps.
|
||||
func (g Guard) Rule(fi fs.FileInfo) (withhold, redact bool) {
|
||||
if g.secret != nil && os.SameFile(g.secret, fi) {
|
||||
return true, false
|
||||
}
|
||||
return false, g.props != nil && os.SameFile(g.props, fi)
|
||||
}
|
||||
|
||||
// ArchiveRule is Rule for an entry of a stored world archive, by its name
|
||||
// cleaned as a path, so "./server.properties" is server.properties too.
|
||||
func ArchiveRule(name string) (withhold, redact bool) {
|
||||
name = path.Clean(name)
|
||||
return name == secretConfigPath, name == propsPath
|
||||
}
|
||||
|
||||
// RedactProps replaces the RCON password in server.properties content with
|
||||
// redactedValue (see redactSecretProps for why a placeholder and not a blank).
|
||||
func RedactProps(content []byte) []byte {
|
||||
return redactSecretProps(propsPath, content)
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
package fileedit
|
||||
|
||||
import "testing"
|
||||
|
||||
// TestArchiveRule: a stored archive's entry is matched by its name as a path,
|
||||
// however the archive spelled it.
|
||||
func TestArchiveRule(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
name string
|
||||
withhold, redact bool
|
||||
}{
|
||||
{"config/paper-global.yml", true, false},
|
||||
{"./config/paper-global.yml", true, false},
|
||||
{"config//paper-global.yml", true, false},
|
||||
{"server.properties", false, true},
|
||||
{"./server.properties", false, true},
|
||||
{"plugins/server.properties", false, false},
|
||||
{"plugins/config/paper-global.yml", false, false},
|
||||
{"config/paper.yml", false, false},
|
||||
} {
|
||||
if w, r := ArchiveRule(c.name); w != c.withhold || r != c.redact {
|
||||
t.Errorf("ArchiveRule(%q) = %v, %v; want %v, %v", c.name, w, r, c.withhold, c.redact)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -25,6 +25,11 @@ const (
|
||||
LabelServer = "felis.lolicon.best/server"
|
||||
LabelOpID = "felis.lolicon.best/files-op"
|
||||
LabelMode = "felis.lolicon.best/files-mode"
|
||||
// LabelAsync marks the Job of an upload or unzip felis-api started and does
|
||||
// not wait on (Editor.StartUpload, Editor.StartUnzip); Ops finds them by it.
|
||||
// AnnotationPath names the file such a Job works on, for Ops to show.
|
||||
LabelAsync = "felis.lolicon.best/files-async"
|
||||
AnnotationPath = "felis.lolicon.best/files-path"
|
||||
|
||||
managedByValue = "felis-files"
|
||||
componentValue = "world-files"
|
||||
@@ -59,7 +64,11 @@ type JobParams struct {
|
||||
UploadSize int64
|
||||
UploadSHA256 string
|
||||
Overwrite bool
|
||||
WorldPVC string
|
||||
// Async marks a Job felis-api does not wait on: it carries LabelAsync and
|
||||
// AnnotationPath, which Ops reads it back by. Only an upload or an unzip
|
||||
// runs so.
|
||||
Async bool
|
||||
WorldPVC string
|
||||
|
||||
Namespace string
|
||||
ServiceAccount string
|
||||
@@ -92,13 +101,17 @@ type JobParams struct {
|
||||
func FilesJobName(server, opID string) string { return "files-" + server + "-" + opID }
|
||||
|
||||
func filesLabels(p JobParams) map[string]string {
|
||||
return map[string]string{
|
||||
l := map[string]string{
|
||||
LabelManagedBy: managedByValue,
|
||||
LabelComponent: componentValue,
|
||||
LabelServer: p.Server,
|
||||
LabelOpID: p.OpID,
|
||||
LabelMode: p.Op,
|
||||
}
|
||||
if p.Async {
|
||||
l[LabelAsync] = "true"
|
||||
}
|
||||
return l
|
||||
}
|
||||
|
||||
// FilesJob renders the file-editor Job. Its isolation is the strictest of the three
|
||||
@@ -152,6 +165,9 @@ func FilesJob(p JobParams) (*batchv1.Job, error) {
|
||||
if p.Op == OpUpload && (p.SourceURL == "" || p.UploadToken == "") {
|
||||
return nil, fmt.Errorf("fileedit: an upload needs a source URL and a token")
|
||||
}
|
||||
if p.Async && p.Op != OpUpload && p.Op != OpUnzip {
|
||||
return nil, fmt.Errorf("fileedit: only an upload or an unzip runs in the background, not %s", p.Op)
|
||||
}
|
||||
limits, err := resourceLimits(p.CPULimit, p.MemLimit)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -195,6 +211,10 @@ func FilesJob(p JobParams) (*batchv1.Job, error) {
|
||||
if p.Overwrite {
|
||||
args = append(args, "--overwrite")
|
||||
}
|
||||
case OpUnzip:
|
||||
if p.Overwrite {
|
||||
args = append(args, "--overwrite")
|
||||
}
|
||||
}
|
||||
container := corev1.Container{
|
||||
Name: containerName,
|
||||
@@ -235,9 +255,10 @@ func FilesJob(p JobParams) (*batchv1.Job, error) {
|
||||
|
||||
job := &batchv1.Job{
|
||||
ObjectMeta: metav1.ObjectMeta{
|
||||
Name: FilesJobName(p.Server, p.OpID),
|
||||
Namespace: p.Namespace,
|
||||
Labels: filesLabels(p),
|
||||
Name: FilesJobName(p.Server, p.OpID),
|
||||
Namespace: p.Namespace,
|
||||
Labels: filesLabels(p),
|
||||
Annotations: filesAnnotations(p),
|
||||
},
|
||||
Spec: batchv1.JobSpec{
|
||||
// One shot: a file operation that failed must surface its failure, not be
|
||||
@@ -297,12 +318,12 @@ func int64Ptr(i int64) *int64 { return &i }
|
||||
|
||||
// filesCapabilities is what the root executor keeps after dropping ALL (see
|
||||
// Config.RunAsUser). DAC_OVERRIDE opens a mode-0600 file (level.dat) the game wrote
|
||||
// as its own uid, which a fixed non-root uid could not. A write, mkdir or upload
|
||||
// also keeps CHOWN so what it creates can be handed to naming.GameUID (exec.go
|
||||
// ownWritten). List, read, delete and rename create nothing and get no more than
|
||||
// they need.
|
||||
// as its own uid, which a fixed non-root uid could not. A write, mkdir, upload or
|
||||
// unzip also keeps CHOWN so what it creates can be handed to naming.GameUID
|
||||
// (exec.go ownWritten). List, read, delete and rename create nothing and get no
|
||||
// more than they need.
|
||||
func filesCapabilities(op string) []corev1.Capability {
|
||||
if op == OpWrite || op == OpMkdir || op == OpUpload {
|
||||
if op == OpWrite || op == OpMkdir || op == OpUpload || op == OpUnzip {
|
||||
return []corev1.Capability{"CHOWN", "DAC_OVERRIDE"}
|
||||
}
|
||||
return []corev1.Capability{"DAC_OVERRIDE"}
|
||||
@@ -323,3 +344,12 @@ func filesPodSecurityContext(p JobParams) *corev1.PodSecurityContext {
|
||||
}
|
||||
return sc
|
||||
}
|
||||
|
||||
// filesAnnotations names the file an async Job works on. A path does not fit a
|
||||
// label (63 characters, no slashes), so it rides an annotation.
|
||||
func filesAnnotations(p JobParams) map[string]string {
|
||||
if !p.Async {
|
||||
return nil
|
||||
}
|
||||
return map[string]string{AnnotationPath: p.Path}
|
||||
}
|
||||
@@ -128,7 +128,7 @@ func TestFilesJobIsolation(t *testing.T) {
|
||||
chown bool
|
||||
}{
|
||||
{OpList, false}, {OpRead, false}, {OpDelete, false}, {OpRename, false},
|
||||
{OpWrite, true}, {OpMkdir, true}, {OpUpload, true},
|
||||
{OpWrite, true}, {OpMkdir, true}, {OpUpload, true}, {OpUnzip, true},
|
||||
} {
|
||||
j, err := FilesJob(opParams(tc.op))
|
||||
if err != nil {
|
||||
@@ -216,6 +216,7 @@ func TestFilesJobWorldMountIsReadOnlyForReads(t *testing.T) {
|
||||
{OpDelete, false},
|
||||
{OpRename, false},
|
||||
{OpUpload, false},
|
||||
{OpUnzip, false},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.op, func(t *testing.T) {
|
||||
@@ -435,6 +436,9 @@ func TestFilesJobRejectsBadParams(t *testing.T) {
|
||||
p.Op, p.Content = OpWrite, make([]byte, MaxWriteBytes+1)
|
||||
}},
|
||||
{"bad cpu limit", func(p *JobParams) { p.CPULimit = "half" }},
|
||||
{"a read in the background", func(p *JobParams) { p.Async = true }},
|
||||
{"a write in the background", func(p *JobParams) { p.Op, p.Async = OpWrite, true }},
|
||||
{"a delete in the background", func(p *JobParams) { p.Op, p.Async = OpDelete, true }},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
@@ -446,3 +450,36 @@ func TestFilesJobRejectsBadParams(t *testing.T) {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestFilesJobAsync checks a background Job is marked so Ops finds it, on the
|
||||
// Job and on its Pod, and carries the path Ops shows; a Job felis-api waits on
|
||||
// carries neither, so Ops never reports it.
|
||||
func TestFilesJobAsync(t *testing.T) {
|
||||
for _, op := range []string{OpUpload, OpUnzip} {
|
||||
t.Run(op, func(t *testing.T) {
|
||||
p := opParams(op)
|
||||
p.Path, p.Async = "maps/world.zip", true
|
||||
j, err := FilesJob(p)
|
||||
if err != nil {
|
||||
t.Fatalf("FilesJob: %v", err)
|
||||
}
|
||||
if j.Labels[LabelAsync] != "true" || j.Spec.Template.Labels[LabelAsync] != "true" {
|
||||
t.Fatalf("job labels %v, pod labels %v, want %s=true on both", j.Labels, j.Spec.Template.Labels, LabelAsync)
|
||||
}
|
||||
if len(j.Annotations) != 1 || j.Annotations[AnnotationPath] != "maps/world.zip" {
|
||||
t.Fatalf("annotations = %v, want only %s", j.Annotations, AnnotationPath)
|
||||
}
|
||||
|
||||
p.Async = false
|
||||
j, err = FilesJob(p)
|
||||
if err != nil {
|
||||
t.Fatalf("FilesJob: %v", err)
|
||||
}
|
||||
_, onJob := j.Labels[LabelAsync]
|
||||
_, onPod := j.Spec.Template.Labels[LabelAsync]
|
||||
if onJob || onPod || j.Annotations != nil {
|
||||
t.Fatalf("a Job waited on is labelled %v / %v and annotated %v", j.Labels, j.Spec.Template.Labels, j.Annotations)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -3,12 +3,15 @@ package fileedit
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
batchv1 "k8s.io/api/batch/v1"
|
||||
corev1 "k8s.io/api/core/v1"
|
||||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
||||
"k8s.io/client-go/kubernetes"
|
||||
@@ -185,3 +188,133 @@ func tail(log string) string {
|
||||
}
|
||||
return "..." + log[len(log)-n:]
|
||||
}
|
||||
|
||||
// Start creates the Job for p and returns without waiting on it. Ops reads it
|
||||
// back.
|
||||
func (k *K8sRunner) Start(ctx context.Context, p JobParams) error {
|
||||
job, err := FilesJob(p)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := k.cs.BatchV1().Jobs(p.Namespace).Create(ctx, job, metav1.CreateOptions{}); err != nil {
|
||||
return fmt.Errorf("fileedit: create file job: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// maxOps bounds what Ops reports, and so how many Pod logs one call reads. Only
|
||||
// one background op runs per server at a time (it holds the world volume), so
|
||||
// this many are the one running and the latest that finished within AsyncTTL.
|
||||
const maxOps = 10
|
||||
|
||||
// opLogLines is how much of an op's log Ops reads: the result line is the last
|
||||
// thing the Job prints to stdout, the progress lines come before it, and a
|
||||
// failed run ends with one line on stderr.
|
||||
const opLogLines = 20
|
||||
|
||||
// Ops lists the server's background Jobs, newest first, with how far each has
|
||||
// got and how it ended, read from the tail of its Pod's log. A Pod whose log
|
||||
// cannot be read yet (still pulling its image) or any more (its node went away)
|
||||
// reports no progress rather than failing the listing.
|
||||
func (k *K8sRunner) Ops(ctx context.Context, namespace, server string) ([]OpState, error) {
|
||||
sel := metav1.ListOptions{LabelSelector: LabelManagedBy + "=" + managedByValue + "," +
|
||||
LabelServer + "=" + server + "," + LabelAsync + "=true"}
|
||||
jobs, err := k.cs.BatchV1().Jobs(namespace).List(ctx, sel)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("fileedit: list file jobs: %w", err)
|
||||
}
|
||||
pods, err := k.cs.CoreV1().Pods(namespace).List(ctx, sel)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("fileedit: list file job pods: %w", err)
|
||||
}
|
||||
// backoffLimit is 0, so a Job has one Pod; the newest wins all the same.
|
||||
podOf := map[string]*corev1.Pod{}
|
||||
for i := range pods.Items {
|
||||
pod := &pods.Items[i]
|
||||
id := pod.Labels[LabelOpID]
|
||||
if cur := podOf[id]; cur == nil || pod.CreationTimestamp.After(cur.CreationTimestamp.Time) {
|
||||
podOf[id] = pod
|
||||
}
|
||||
}
|
||||
items := jobs.Items
|
||||
sort.SliceStable(items, func(i, j int) bool {
|
||||
return items[i].CreationTimestamp.After(items[j].CreationTimestamp.Time)
|
||||
})
|
||||
if len(items) > maxOps {
|
||||
items = items[:maxOps]
|
||||
}
|
||||
out := make([]OpState, 0, len(items))
|
||||
for i := range items {
|
||||
log := ""
|
||||
if pod := podOf[items[i].Labels[LabelOpID]]; pod != nil && pod.Status.Phase != corev1.PodPending {
|
||||
log, _ = k.logTail(ctx, namespace, pod.Name)
|
||||
}
|
||||
out = append(out, opState(&items[i], log))
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// logTail reads the last opLogLines lines of a Pod's log.
|
||||
func (k *K8sRunner) logTail(ctx context.Context, namespace, pod string) (string, error) {
|
||||
lines := int64(opLogLines)
|
||||
stream, err := k.cs.CoreV1().Pods(namespace).GetLogs(pod, &corev1.PodLogOptions{
|
||||
Container: containerName, TailLines: &lines,
|
||||
}).Stream(ctx)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer stream.Close()
|
||||
b, err := io.ReadAll(io.LimitReader(stream, maxLogBytes))
|
||||
return string(b), err
|
||||
}
|
||||
|
||||
// opState reads one background Job, and the tail of its Pod's log, as an
|
||||
// OpState. The printed result decides the outcome whatever the Job's condition
|
||||
// says: a Job killed at its deadline just after printing did finish its work.
|
||||
// A Job that ended without one failed, and the condition's reason says how
|
||||
// (DeadlineExceeded, BackoffLimitExceeded); a Job that completed but whose log
|
||||
// could not be read has an outcome no one can tell, ResultUnavailable.
|
||||
func opState(job *batchv1.Job, log string) OpState {
|
||||
st := OpState{
|
||||
ID: job.Labels[LabelOpID], Op: job.Labels[LabelMode], Path: job.Annotations[AnnotationPath],
|
||||
State: OpRunning, Started: job.CreationTimestamp.Time,
|
||||
}
|
||||
if p, ok := lastProgress(log); ok {
|
||||
st.Done, st.Total = p.Done, p.Total
|
||||
}
|
||||
ended, reason := false, ""
|
||||
for _, c := range job.Status.Conditions {
|
||||
if c.Status != corev1.ConditionTrue {
|
||||
continue
|
||||
}
|
||||
switch c.Type {
|
||||
case batchv1.JobComplete, batchv1.JobSuccessCriteriaMet:
|
||||
ended, reason = true, "ResultUnavailable"
|
||||
st.Finished = c.LastTransitionTime.Time
|
||||
case batchv1.JobFailed, batchv1.JobFailureTarget:
|
||||
ended, reason = true, c.Reason
|
||||
st.Finished = c.LastTransitionTime.Time
|
||||
}
|
||||
}
|
||||
if !ended {
|
||||
return st
|
||||
}
|
||||
if payload, ok := extractResult(log); ok {
|
||||
var res Result
|
||||
if json.Unmarshal(payload, &res) == nil {
|
||||
st.Result = &res
|
||||
}
|
||||
}
|
||||
switch {
|
||||
case st.Result != nil && st.Result.Code == "":
|
||||
st.State = OpSucceeded
|
||||
case st.Result != nil:
|
||||
st.State = OpFailed
|
||||
default:
|
||||
st.State, st.Reason = OpFailed, reason
|
||||
if st.Reason == "" {
|
||||
st.Reason = "Failed"
|
||||
}
|
||||
}
|
||||
return st
|
||||
}
|
||||
@@ -0,0 +1,241 @@
|
||||
package fileedit
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
batchv1 "k8s.io/api/batch/v1"
|
||||
corev1 "k8s.io/api/core/v1"
|
||||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
||||
"k8s.io/apimachinery/pkg/runtime"
|
||||
"k8s.io/client-go/kubernetes/fake"
|
||||
k8stesting "k8s.io/client-go/testing"
|
||||
)
|
||||
|
||||
var (
|
||||
opCreated = time.Date(2026, 9, 28, 10, 0, 0, 0, time.UTC)
|
||||
opEnded = opCreated.Add(3 * time.Minute)
|
||||
)
|
||||
|
||||
func asyncJob(conds ...batchv1.JobCondition) *batchv1.Job {
|
||||
return &batchv1.Job{
|
||||
ObjectMeta: metav1.ObjectMeta{
|
||||
Name: "files-survival-0a",
|
||||
CreationTimestamp: metav1.NewTime(opCreated),
|
||||
Labels: map[string]string{LabelOpID: "0a", LabelMode: OpUnzip},
|
||||
Annotations: map[string]string{AnnotationPath: "maps/world.zip"},
|
||||
},
|
||||
Status: batchv1.JobStatus{Conditions: conds},
|
||||
}
|
||||
}
|
||||
|
||||
func cond(typ batchv1.JobConditionType, status corev1.ConditionStatus, reason string) batchv1.JobCondition {
|
||||
return batchv1.JobCondition{Type: typ, Status: status, Reason: reason, LastTransitionTime: metav1.NewTime(opEnded)}
|
||||
}
|
||||
|
||||
// TestOpState pins how a background Job and the tail of its log read as an
|
||||
// OpState: the printed result decides the outcome whatever the Job's condition
|
||||
// says, and a Job that ended without one failed for the condition's reason.
|
||||
func TestOpState(t *testing.T) {
|
||||
progress := ProgressPrefix + `{"done":10,"total":100}` + "\n" +
|
||||
"a stderr line\n" +
|
||||
ProgressPrefix + `{"done":40,"total":100}` + "\n"
|
||||
ok := ResultPrefix + `{"files":3,"bytes":40}` + "\n"
|
||||
conflict := ResultPrefix + `{"code":"exists","conflicts":["a.txt"],"conflict_count":1}` + "\n"
|
||||
complete := cond(batchv1.JobComplete, corev1.ConditionTrue, "")
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
conds []batchv1.JobCondition
|
||||
log string
|
||||
state string
|
||||
reason string
|
||||
code string
|
||||
files int
|
||||
done int64
|
||||
finished bool
|
||||
wantResult bool
|
||||
}{
|
||||
{name: "running, at its latest progress", log: progress, state: OpRunning, done: 40},
|
||||
{name: "running, before any progress", state: OpRunning},
|
||||
{name: "a condition not yet true is still running",
|
||||
conds: []batchv1.JobCondition{cond(batchv1.JobFailed, corev1.ConditionFalse, "DeadlineExceeded")},
|
||||
log: progress, state: OpRunning, done: 40},
|
||||
{name: "complete with a clean result",
|
||||
conds: []batchv1.JobCondition{complete}, log: progress + ok,
|
||||
state: OpSucceeded, files: 3, done: 40, finished: true, wantResult: true},
|
||||
{name: "success criteria met before complete",
|
||||
conds: []batchv1.JobCondition{cond(batchv1.JobSuccessCriteriaMet, corev1.ConditionTrue, "")}, log: ok,
|
||||
state: OpSucceeded, files: 3, finished: true, wantResult: true},
|
||||
{name: "killed at its deadline after printing a clean result",
|
||||
conds: []batchv1.JobCondition{cond(batchv1.JobFailed, corev1.ConditionTrue, "DeadlineExceeded")}, log: ok,
|
||||
state: OpSucceeded, files: 3, finished: true, wantResult: true},
|
||||
{name: "complete with a refusal",
|
||||
conds: []batchv1.JobCondition{complete}, log: conflict,
|
||||
state: OpFailed, code: CodeExists, finished: true, wantResult: true},
|
||||
{name: "killed at its deadline without a result",
|
||||
conds: []batchv1.JobCondition{cond(batchv1.JobFailed, corev1.ConditionTrue, "DeadlineExceeded")}, log: progress,
|
||||
state: OpFailed, reason: "DeadlineExceeded", done: 40, finished: true},
|
||||
{name: "failure target before failed",
|
||||
conds: []batchv1.JobCondition{cond(batchv1.JobFailureTarget, corev1.ConditionTrue, "BackoffLimitExceeded")},
|
||||
state: OpFailed, reason: "BackoffLimitExceeded", finished: true},
|
||||
{name: "failed without a reason",
|
||||
conds: []batchv1.JobCondition{cond(batchv1.JobFailed, corev1.ConditionTrue, "")},
|
||||
state: OpFailed, reason: "Failed", finished: true},
|
||||
{name: "complete but its log is gone",
|
||||
conds: []batchv1.JobCondition{complete},
|
||||
state: OpFailed, reason: "ResultUnavailable", finished: true},
|
||||
{name: "complete with a result that does not parse",
|
||||
conds: []batchv1.JobCondition{complete}, log: ResultPrefix + "{\n",
|
||||
state: OpFailed, reason: "ResultUnavailable", finished: true},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
st := opState(asyncJob(tc.conds...), tc.log)
|
||||
if st.ID != "0a" || st.Op != OpUnzip || st.Path != "maps/world.zip" || !st.Started.Equal(opCreated) {
|
||||
t.Fatalf("identity = %q %q %q %v", st.ID, st.Op, st.Path, st.Started)
|
||||
}
|
||||
if st.State != tc.state || st.Reason != tc.reason {
|
||||
t.Fatalf("state = %q reason %q, want %q reason %q", st.State, st.Reason, tc.state, tc.reason)
|
||||
}
|
||||
if st.Done != tc.done || (tc.done != 0 && st.Total != 100) {
|
||||
t.Fatalf("progress = %d/%d, want %d/100", st.Done, st.Total, tc.done)
|
||||
}
|
||||
if want := map[bool]time.Time{true: opEnded}[tc.finished]; !st.Finished.Equal(want) {
|
||||
t.Fatalf("finished = %v, want %v", st.Finished, want)
|
||||
}
|
||||
if (st.Result != nil) != tc.wantResult {
|
||||
t.Fatalf("result = %+v, want one: %v", st.Result, tc.wantResult)
|
||||
}
|
||||
if st.Result != nil && (st.Result.Code != tc.code || st.Result.Files != tc.files) {
|
||||
t.Fatalf("result = %+v, want code %q and %d files", st.Result, tc.code, tc.files)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestK8sRunnerOps checks what Ops lists: this server's background Jobs only,
|
||||
// newest first and at most maxOps of them, with a log read for each started Pod
|
||||
// and none for a Pod still waiting to run.
|
||||
func TestK8sRunnerOps(t *testing.T) {
|
||||
job := func(server, id string, age time.Duration, async bool) *batchv1.Job {
|
||||
p := testParams(OpUnzip)
|
||||
p.Server, p.OpID, p.Path, p.Async = server, id, "maps/"+id+".zip", async
|
||||
j, err := FilesJob(p)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
j.CreationTimestamp = metav1.NewTime(opCreated.Add(-age))
|
||||
return j
|
||||
}
|
||||
pod := func(j *batchv1.Job, phase corev1.PodPhase, age time.Duration) *corev1.Pod {
|
||||
return &corev1.Pod{
|
||||
ObjectMeta: metav1.ObjectMeta{
|
||||
Name: fmt.Sprintf("%s-%d", j.Name, age), Namespace: "minecraft", Labels: j.Spec.Template.Labels,
|
||||
CreationTimestamp: metav1.NewTime(opCreated.Add(-age)),
|
||||
},
|
||||
Status: corev1.PodStatus{Phase: phase},
|
||||
}
|
||||
}
|
||||
|
||||
var objs []runtime.Object
|
||||
for i := range maxOps + 2 {
|
||||
j := job("survival", fmt.Sprintf("%02d", i), time.Duration(i)*time.Minute, true)
|
||||
objs = append(objs, j, pod(j, corev1.PodSucceeded, time.Duration(i)*time.Minute))
|
||||
}
|
||||
// The newest Job's newest Pod has not started, so its log is not read; the
|
||||
// older Pod beside it is not the one Ops reports on.
|
||||
newest := job("survival", "new", -time.Minute, true)
|
||||
objs = append(objs, newest,
|
||||
pod(newest, corev1.PodPending, -time.Minute), pod(newest, corev1.PodFailed, 0))
|
||||
objs = append(objs,
|
||||
job("creative", "other", -2*time.Minute, true),
|
||||
job("survival", "sync", -3*time.Minute, false))
|
||||
cs := fake.NewSimpleClientset(objs...)
|
||||
|
||||
ops, err := NewK8sRunner(cs).Ops(context.Background(), "minecraft", "survival")
|
||||
if err != nil {
|
||||
t.Fatalf("Ops: %v", err)
|
||||
}
|
||||
var ids []string
|
||||
for _, op := range ops {
|
||||
ids = append(ids, op.ID)
|
||||
}
|
||||
want := []string{"new", "00", "01", "02", "03", "04", "05", "06", "07", "08"}
|
||||
if fmt.Sprint(ids) != fmt.Sprint(want) {
|
||||
t.Fatalf("ops = %v, want %v", ids, want)
|
||||
}
|
||||
if ops[0].Path != "maps/new.zip" || ops[0].Op != OpUnzip {
|
||||
t.Fatalf("newest op = %+v", ops[0])
|
||||
}
|
||||
|
||||
logs := 0
|
||||
for _, a := range cs.Actions() {
|
||||
if a.GetVerb() == "get" && a.GetSubresource() == "log" {
|
||||
logs++
|
||||
opts := a.(k8stesting.GenericAction).GetValue().(*corev1.PodLogOptions)
|
||||
if opts.Container != containerName || opts.TailLines == nil || *opts.TailLines != opLogLines {
|
||||
t.Fatalf("log options = %+v", opts)
|
||||
}
|
||||
}
|
||||
}
|
||||
if logs != maxOps-1 {
|
||||
t.Fatalf("read %d logs, want %d: one per listed op whose Pod has started", logs, maxOps-1)
|
||||
}
|
||||
}
|
||||
|
||||
// TestK8sRunnerOpsFailsLoudly checks a listing the cluster refused is an error,
|
||||
// never an empty list that would read as nothing running.
|
||||
func TestK8sRunnerOpsFailsLoudly(t *testing.T) {
|
||||
for _, resource := range []string{"jobs", "pods"} {
|
||||
t.Run(resource, func(t *testing.T) {
|
||||
cs := fake.NewSimpleClientset()
|
||||
cs.PrependReactor("list", resource, func(k8stesting.Action) (bool, runtime.Object, error) {
|
||||
return true, nil, fmt.Errorf("forbidden")
|
||||
})
|
||||
ops, err := NewK8sRunner(cs).Ops(context.Background(), "minecraft", "survival")
|
||||
if err == nil || ops != nil {
|
||||
t.Fatalf("Ops = %v, %v; want the refusal", ops, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestK8sRunnerStart checks Start creates the rendered Job and nothing else,
|
||||
// and refuses params the renderer refuses without touching the cluster.
|
||||
func TestK8sRunnerStart(t *testing.T) {
|
||||
cs := fake.NewSimpleClientset()
|
||||
p := testParams(OpUnzip)
|
||||
p.Path, p.Async = "maps/world.zip", true
|
||||
if err := NewK8sRunner(cs).Start(context.Background(), p); err != nil {
|
||||
t.Fatalf("Start: %v", err)
|
||||
}
|
||||
want, _ := FilesJob(p)
|
||||
got, err := cs.BatchV1().Jobs("minecraft").Get(context.Background(), want.Name, metav1.GetOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("the Job was not created: %v", err)
|
||||
}
|
||||
if got.Labels[LabelAsync] != "true" || got.Annotations[AnnotationPath] != "maps/world.zip" {
|
||||
t.Fatalf("created %+v", got.ObjectMeta)
|
||||
}
|
||||
if n := len(cs.Actions()); n != 2 { // the create, and this test's get
|
||||
t.Fatalf("%d calls to the cluster, want the one create", n-1)
|
||||
}
|
||||
|
||||
cs = fake.NewSimpleClientset()
|
||||
p.Op = OpRead
|
||||
if err := NewK8sRunner(cs).Start(context.Background(), p); err == nil || len(cs.Actions()) != 0 {
|
||||
t.Fatalf("err %v after %d calls, want a refusal before any", err, len(cs.Actions()))
|
||||
}
|
||||
|
||||
cs = fake.NewSimpleClientset()
|
||||
cs.PrependReactor("create", "jobs", func(k8stesting.Action) (bool, runtime.Object, error) {
|
||||
return true, nil, fmt.Errorf("quota exceeded")
|
||||
})
|
||||
p.Op = OpUnzip
|
||||
if err := NewK8sRunner(cs).Start(context.Background(), p); err == nil {
|
||||
t.Fatal("a refused create must be an error")
|
||||
}
|
||||
}
|
||||
@@ -491,17 +491,39 @@ func TestUpload(t *testing.T) {
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("over the cap is too_large and never fetched; at the cap is fetched", func(t *testing.T) {
|
||||
t.Run("more than the volume has free is no_space and never fetched; exactly the free room is fetched", func(t *testing.T) {
|
||||
root, _ := worldRoot(t)
|
||||
stubStatfs(t, uint64(len(jar)), 1<<30)
|
||||
src := &fakeSource{body: jar}
|
||||
res, err := send(t, root, "big.jar", src.upload(MaxUploadBytes+1, ""), false)
|
||||
if err != nil || res.Code != CodeTooLarge || src.opened != 0 {
|
||||
t.Fatalf("result = %+v, %v, opened %d", res, err, src.opened)
|
||||
res, err := send(t, root, "big.jar", src.upload(int64(len(jar))+1, ""), false)
|
||||
if err != nil || res.Code != CodeNoSpace || res.Need != int64(len(jar))+1 || res.Avail != int64(len(jar)) || src.opened != 0 {
|
||||
t.Fatalf("result = %+v, %v, opened %d; want no_space with need %d, avail %d", res, err, src.opened, len(jar)+1, len(jar))
|
||||
}
|
||||
// At the cap the size passes and the transfer starts; this source then
|
||||
// comes up short, which is a broken transfer rather than a refusal.
|
||||
if _, err := send(t, root, "big.jar", src.upload(MaxUploadBytes, ""), false); err == nil || src.opened != 1 {
|
||||
t.Fatalf("at the cap: err = %v, opened %d; want a fetch", err, src.opened)
|
||||
assertAbsent(t, filepath.Join(root, "big.jar"))
|
||||
if res, err := send(t, root, "big.jar", whole(src), false); err != nil || res.Code != "" || src.opened != 1 {
|
||||
t.Fatalf("at the free room: result = %+v, %v, opened %d; want it landed", res, err, src.opened)
|
||||
}
|
||||
})
|
||||
|
||||
// The editor used to stop at MaxUploadBytes; a file sent in parts is bounded
|
||||
// by the volume alone.
|
||||
t.Run("past the single-request limit is fetched", func(t *testing.T) {
|
||||
root, _ := worldRoot(t)
|
||||
stubStatfs(t, 1<<40, 1<<41)
|
||||
src := &fakeSource{body: jar}
|
||||
// This source then comes up short, which is a broken transfer, not a refusal.
|
||||
if _, err := send(t, root, "big.jar", src.upload(MaxUploadBytes+1, ""), false); err == nil || src.opened != 1 {
|
||||
t.Fatalf("err = %v, opened %d; want a fetch", err, src.opened)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("progress hears every byte", func(t *testing.T) {
|
||||
root, _ := worldRoot(t)
|
||||
var last, calls, total int64
|
||||
res := exec(t, root, Request{Op: OpUpload, Path: "x.jar", Upload: whole(&fakeSource{body: jar}),
|
||||
Progress: func(done, all int64) { calls++; last, total = done, all }})
|
||||
if res.Code != "" || calls == 0 || last != int64(len(jar)) || total != int64(len(jar)) {
|
||||
t.Fatalf("result = %+v; progress calls %d, last %d of %d; want the whole %d", res, calls, last, total, len(jar))
|
||||
}
|
||||
})
|
||||
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
package fileedit
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ProgressPrefix marks a progress line: how many of an upload's or an unzip's
|
||||
// bytes are in so far, as JSON. Those two run as Jobs felis-api does not wait on,
|
||||
// and the panel shows how far one has got by reading the latest such line from
|
||||
// the tail of the Pod's log. Like ResultPrefix it is found by its marker, since
|
||||
// the log is stdout and stderr merged.
|
||||
const ProgressPrefix = "FELIS-FILES-PROGRESS: "
|
||||
|
||||
// Progress is one progress line.
|
||||
type Progress struct {
|
||||
Done int64 `json:"done"`
|
||||
Total int64 `json:"total"`
|
||||
}
|
||||
|
||||
// ThrottledProgress returns a progress func that prints to w at most once per
|
||||
// every, plus the first call (the last print starts at the zero time) and the
|
||||
// one that reaches the total, so the log grows by a line a second however fast
|
||||
// the bytes move and still ends on the true final count. A write error is
|
||||
// dropped: progress is a courtesy, and the result line that follows is what
|
||||
// felis-api acts on.
|
||||
func ThrottledProgress(w io.Writer, every time.Duration, now func() time.Time) func(done, total int64) {
|
||||
var last time.Time
|
||||
return func(done, total int64) {
|
||||
t := now()
|
||||
if done < total && t.Sub(last) < every {
|
||||
return
|
||||
}
|
||||
last = t
|
||||
b, _ := json.Marshal(Progress{Done: done, Total: total})
|
||||
fmt.Fprintf(w, "%s%s\n", ProgressPrefix, b)
|
||||
}
|
||||
}
|
||||
|
||||
// lastProgress finds the latest well-formed progress line in a log.
|
||||
func lastProgress(log string) (Progress, bool) {
|
||||
var p Progress
|
||||
found := false
|
||||
sc := bufio.NewScanner(strings.NewReader(log))
|
||||
sc.Buffer(make([]byte, 0, 4096), maxLogBytes)
|
||||
for sc.Scan() {
|
||||
rest, ok := strings.CutPrefix(sc.Text(), ProgressPrefix)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
var q Progress
|
||||
if json.Unmarshal([]byte(rest), &q) == nil {
|
||||
p, found = q, true
|
||||
}
|
||||
}
|
||||
return p, found
|
||||
}
|
||||
@@ -0,0 +1,42 @@
|
||||
package fileedit
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestThrottledProgress(t *testing.T) {
|
||||
var out bytes.Buffer
|
||||
clock := time.Unix(1000, 0)
|
||||
progress := ThrottledProgress(&out, time.Second, func() time.Time { return clock })
|
||||
|
||||
progress(0, 100) // first call: printed
|
||||
progress(10, 100) // same instant: dropped
|
||||
clock = clock.Add(999 * time.Millisecond)
|
||||
progress(20, 100) // not a second yet: dropped
|
||||
clock = clock.Add(time.Millisecond)
|
||||
progress(30, 100) // a second on: printed
|
||||
progress(100, 100) // the total, however soon: printed
|
||||
want := ProgressPrefix + `{"done":0,"total":100}` + "\n" +
|
||||
ProgressPrefix + `{"done":30,"total":100}` + "\n" +
|
||||
ProgressPrefix + `{"done":100,"total":100}` + "\n"
|
||||
if out.String() != want {
|
||||
t.Fatalf("printed:\n%s\nwant:\n%s", out.String(), want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLastProgress(t *testing.T) {
|
||||
log := "noise\n" +
|
||||
ProgressPrefix + `{"done":1,"total":9}` + "\n" +
|
||||
"a runtime warning on stderr\n" +
|
||||
ProgressPrefix + `{"done":5,"total":9}` + "\n" +
|
||||
ProgressPrefix + `{"done":` + "\n" // cut off mid-line by the tail
|
||||
p, ok := lastProgress(log)
|
||||
if !ok || p != (Progress{Done: 5, Total: 9}) {
|
||||
t.Fatalf("lastProgress = %+v, %v; want {5 9}, true", p, ok)
|
||||
}
|
||||
if _, ok := lastProgress("no marker here\n"); ok {
|
||||
t.Fatal("a log without a progress line reported one")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,328 @@
|
||||
package fileedit
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"crypto/subtle"
|
||||
"encoding"
|
||||
"errors"
|
||||
"fmt"
|
||||
"hash"
|
||||
"io"
|
||||
"os"
|
||||
"time"
|
||||
)
|
||||
|
||||
// A file too big for one request body arrives as a session: Begin declares its
|
||||
// size and where it goes, Append adds one part at a time in order, and Seal
|
||||
// hands the finished file to the Job that lands it, which fetches it through
|
||||
// Open like any other staged upload. The Cloudflare edge refuses bodies over
|
||||
// 100 MB, so a part is at most PartBytes; the file itself has no ceiling but the
|
||||
// room on the staging disk, and Begin reserves all of it up front, so an upload
|
||||
// that starts is one the disk can finish.
|
||||
//
|
||||
// Every call names the user and the server the session was begun for, and a
|
||||
// session answers no one else: an id that is someone else's reads as unknown.
|
||||
//
|
||||
// A part that fails midway (the connection dropped, the edge cut it off) is
|
||||
// rolled back to where it started, so the session's length is always the resume
|
||||
// point. A sealed session stays until it has been served whole once (Served), so
|
||||
// a Job that failed before it had every byte can be started again without the
|
||||
// file being sent again; one left idle for SessionIdle is dropped (Expire).
|
||||
|
||||
// PartBytes is the largest part Append takes, matching the modpack upload's
|
||||
// parts (submit.DefaultPartMaxBytes).
|
||||
const PartBytes = 32 << 20
|
||||
|
||||
// SessionIdle is how long a session may sit untouched before Expire drops it:
|
||||
// long enough to resume after a lost connection or a laptop lid, short enough
|
||||
// that an abandoned upload gives its room back the same day.
|
||||
const SessionIdle = 6 * time.Hour
|
||||
|
||||
// MaxSessionsPerUser bounds the sessions one user holds open at once. Each
|
||||
// reserves its whole size on the staging disk, so without a bound one user
|
||||
// could reserve the disk out from under everyone for SessionIdle.
|
||||
const MaxSessionsPerUser = 4
|
||||
|
||||
var (
|
||||
// ErrTooManySessions is a Begin by a user who already holds
|
||||
// MaxSessionsPerUser sessions.
|
||||
ErrTooManySessions = errors.New("fileedit: too many uploads in progress")
|
||||
// ErrUploadBusy is a call on a session another request is still appending
|
||||
// to. Parts go one at a time.
|
||||
ErrUploadBusy = errors.New("fileedit: another request is still writing this upload")
|
||||
// ErrPartTooLarge is a part over PartBytes, or one that runs past the size
|
||||
// the session was begun with.
|
||||
ErrPartTooLarge = errors.New("fileedit: the part is too large")
|
||||
// ErrUploadIncomplete is a Seal before every byte has arrived.
|
||||
ErrUploadIncomplete = errors.New("fileedit: the upload has not finished arriving")
|
||||
)
|
||||
|
||||
// OffsetError is a part that does not start where the session ends. Received is
|
||||
// where it does end, so the client resumes from there.
|
||||
type OffsetError struct{ Received int64 }
|
||||
|
||||
func (e *OffsetError) Error() string {
|
||||
return fmt.Sprintf("fileedit: the upload holds %d bytes; send the part that starts there", e.Received)
|
||||
}
|
||||
|
||||
// Session is where one session stands.
|
||||
type Session struct {
|
||||
ID string
|
||||
Path string
|
||||
Size int64
|
||||
Received int64
|
||||
}
|
||||
|
||||
type session struct {
|
||||
user, server, path string
|
||||
file string
|
||||
size, received int64
|
||||
h hash.Hash
|
||||
busy bool
|
||||
touched time.Time
|
||||
|
||||
// armed is set by Seal with the digest of the token it minted and cleared by
|
||||
// the Open that spends it.
|
||||
armed bool
|
||||
tokenHash [sha256.Size]byte
|
||||
}
|
||||
|
||||
func (s *Stage) now() time.Time {
|
||||
if s.Now != nil {
|
||||
return s.Now()
|
||||
}
|
||||
return time.Now()
|
||||
}
|
||||
|
||||
// Begin opens a session for a file of size bytes that will land at path on
|
||||
// server, reserving room for all of it.
|
||||
func (s *Stage) Begin(user, server, path string, size int64) (Session, error) {
|
||||
if size < 0 {
|
||||
return Session{}, fmt.Errorf("fileedit: an upload of %d bytes", size)
|
||||
}
|
||||
id, err := randomHex(16)
|
||||
if err != nil {
|
||||
return Session{}, fmt.Errorf("fileedit: generate an upload id: %w", err)
|
||||
}
|
||||
if err := os.MkdirAll(s.Dir, 0o700); err != nil {
|
||||
return Session{}, fmt.Errorf("fileedit: create the upload stage: %w", err)
|
||||
}
|
||||
if err := s.reserve(size); err != nil {
|
||||
return Session{}, err
|
||||
}
|
||||
f, err := os.CreateTemp(s.Dir, "session-*")
|
||||
if err != nil {
|
||||
s.unreserve(size)
|
||||
return Session{}, fmt.Errorf("fileedit: stage the upload: %w", err)
|
||||
}
|
||||
f.Close()
|
||||
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
n := 0
|
||||
for _, ss := range s.sessions {
|
||||
if ss.user == user {
|
||||
n++
|
||||
}
|
||||
}
|
||||
if n >= MaxSessionsPerUser {
|
||||
s.reserved -= size
|
||||
os.Remove(f.Name())
|
||||
return Session{}, fmt.Errorf("%w: finish or cancel one of your %d uploads first", ErrTooManySessions, n)
|
||||
}
|
||||
if s.sessions == nil {
|
||||
s.sessions = map[string]*session{}
|
||||
}
|
||||
s.sessions[id] = &session{
|
||||
user: user, server: server, path: path, file: f.Name(),
|
||||
size: size, h: sha256.New(), touched: s.now(),
|
||||
}
|
||||
return Session{ID: id, Path: path, Size: size}, nil
|
||||
}
|
||||
|
||||
// lookup finds the caller's session. s.mu must be held.
|
||||
func (s *Stage) lookup(user, server, id string) (*session, error) {
|
||||
ss, ok := s.sessions[id]
|
||||
if !ok || ss.user != user || ss.server != server {
|
||||
return nil, ErrNotStaged
|
||||
}
|
||||
return ss, nil
|
||||
}
|
||||
|
||||
func (ss *session) view(id string) Session {
|
||||
return Session{ID: id, Path: ss.path, Size: ss.size, Received: ss.received}
|
||||
}
|
||||
|
||||
// Status reports where the caller's session stands.
|
||||
func (s *Stage) Status(user, server, id string) (Session, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
ss, err := s.lookup(user, server, id)
|
||||
if err != nil {
|
||||
return Session{}, err
|
||||
}
|
||||
return ss.view(id), nil
|
||||
}
|
||||
|
||||
// Append adds the n bytes of body at offset, which must be where the session
|
||||
// ends. body must end right after them (an HTTP body of that Content-Length
|
||||
// does). On any failure the session is left as it was before the call.
|
||||
func (s *Stage) Append(user, server, id string, offset int64, body io.Reader, n int64) (Session, error) {
|
||||
s.mu.Lock()
|
||||
ss, err := s.lookup(user, server, id)
|
||||
switch {
|
||||
case err != nil:
|
||||
case ss.busy:
|
||||
err = ErrUploadBusy
|
||||
case offset != ss.received:
|
||||
err = &OffsetError{Received: ss.received}
|
||||
case n < 0 || n > PartBytes || n > ss.size-ss.received:
|
||||
err = fmt.Errorf("%w: %d bytes at %d of a %d-byte upload; parts are at most %d bytes",
|
||||
ErrPartTooLarge, n, offset, ss.size, PartBytes)
|
||||
}
|
||||
if err != nil {
|
||||
var view Session
|
||||
if ss != nil {
|
||||
view = ss.view(id)
|
||||
}
|
||||
s.mu.Unlock()
|
||||
return view, err
|
||||
}
|
||||
ss.busy = true
|
||||
s.mu.Unlock()
|
||||
|
||||
// The session is ours until busy is cleared, so the file and the hash are
|
||||
// touched without the lock. The hash's state is kept to undo a failed part.
|
||||
before, err := ss.h.(encoding.BinaryMarshaler).MarshalBinary()
|
||||
if err == nil {
|
||||
err = appendPart(ss.file, offset, body, n, ss.h)
|
||||
if err != nil {
|
||||
_ = os.Truncate(ss.file, offset)
|
||||
_ = ss.h.(encoding.BinaryUnmarshaler).UnmarshalBinary(before)
|
||||
}
|
||||
}
|
||||
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
ss.busy = false
|
||||
ss.touched = s.now()
|
||||
if err != nil {
|
||||
return ss.view(id), err
|
||||
}
|
||||
ss.received += n
|
||||
s.reserved -= n
|
||||
return ss.view(id), nil
|
||||
}
|
||||
|
||||
// appendPart writes exactly n bytes of body at offset in the file named file,
|
||||
// feeding them to h as well.
|
||||
func appendPart(file string, offset int64, body io.Reader, n int64, h hash.Hash) error {
|
||||
f, err := os.OpenFile(file, os.O_WRONLY, 0)
|
||||
if err != nil {
|
||||
return fmt.Errorf("fileedit: open the staged upload: %w", err)
|
||||
}
|
||||
src := &bodyReader{r: body}
|
||||
// One byte past n, so the read that finds the end happens here.
|
||||
got, copyErr := io.Copy(io.MultiWriter(io.NewOffsetWriter(f, offset), h), io.LimitReader(src, n+1))
|
||||
closeErr := f.Close()
|
||||
return stageFailure(src.err, copyErr, closeErr, got, n)
|
||||
}
|
||||
|
||||
// Seal ends the caller's session and arms it for one fetch: the Staged it
|
||||
// returns carries a fresh token, and any token an earlier Seal minted stops
|
||||
// working. Every byte must have arrived.
|
||||
func (s *Stage) Seal(user, server, id string) (Staged, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
ss, err := s.lookup(user, server, id)
|
||||
if err != nil {
|
||||
return Staged{}, err
|
||||
}
|
||||
if ss.busy {
|
||||
return Staged{}, ErrUploadBusy
|
||||
}
|
||||
if ss.received != ss.size {
|
||||
return Staged{}, fmt.Errorf("%w: %d of %d bytes are here", ErrUploadIncomplete, ss.received, ss.size)
|
||||
}
|
||||
st, tokenHash, err := newHandle(ss.h, ss.size)
|
||||
if err != nil {
|
||||
return Staged{}, err
|
||||
}
|
||||
st.ID = id
|
||||
ss.armed, ss.tokenHash = true, tokenHash
|
||||
ss.touched = s.now()
|
||||
return st, nil
|
||||
}
|
||||
|
||||
// openSession is Open for a sealed session. s.mu must be held; ok is false when
|
||||
// id names no session.
|
||||
func (s *Stage) openSession(id string, sum [sha256.Size]byte) (path string, size int64, ok bool, err error) {
|
||||
ss, found := s.sessions[id]
|
||||
if !found {
|
||||
return "", 0, false, nil
|
||||
}
|
||||
if !ss.armed || subtle.ConstantTimeCompare(sum[:], ss.tokenHash[:]) != 1 {
|
||||
return "", 0, true, ErrNotStaged
|
||||
}
|
||||
ss.armed = false
|
||||
ss.touched = s.now()
|
||||
return ss.file, ss.size, true, nil
|
||||
}
|
||||
|
||||
// Served tells the stage the session id was sent whole to the Job that opened
|
||||
// it, and deletes it: its bytes are on the Job's side now. An id that names no
|
||||
// session (an upload staged by Put, which its own release deletes) is ignored.
|
||||
func (s *Stage) Served(id string) {
|
||||
s.mu.Lock()
|
||||
ss, ok := s.sessions[id]
|
||||
if ok {
|
||||
delete(s.sessions, id)
|
||||
}
|
||||
s.mu.Unlock()
|
||||
if ok {
|
||||
os.Remove(ss.file)
|
||||
}
|
||||
}
|
||||
|
||||
// Drop cancels the caller's session and deletes what it holds.
|
||||
func (s *Stage) Drop(user, server, id string) error {
|
||||
s.mu.Lock()
|
||||
ss, err := s.lookup(user, server, id)
|
||||
if err == nil && ss.busy {
|
||||
err = ErrUploadBusy
|
||||
}
|
||||
if err != nil {
|
||||
s.mu.Unlock()
|
||||
return err
|
||||
}
|
||||
s.dropLocked(id, ss)
|
||||
s.mu.Unlock()
|
||||
os.Remove(ss.file)
|
||||
return nil
|
||||
}
|
||||
|
||||
// dropLocked forgets a session and gives back the room it still had reserved.
|
||||
// s.mu must be held; the caller deletes the file.
|
||||
func (s *Stage) dropLocked(id string, ss *session) {
|
||||
delete(s.sessions, id)
|
||||
s.reserved -= ss.size - ss.received
|
||||
}
|
||||
|
||||
// Expire drops every session untouched for SessionIdle, sealed or not, and
|
||||
// reports how many it dropped. A session a part is arriving for is never idle.
|
||||
func (s *Stage) Expire() int {
|
||||
cutoff := s.now().Add(-SessionIdle)
|
||||
var files []string
|
||||
s.mu.Lock()
|
||||
for id, ss := range s.sessions {
|
||||
if !ss.busy && ss.touched.Before(cutoff) {
|
||||
s.dropLocked(id, ss)
|
||||
files = append(files, ss.file)
|
||||
}
|
||||
}
|
||||
s.mu.Unlock()
|
||||
for _, f := range files {
|
||||
os.Remove(f)
|
||||
}
|
||||
return len(files)
|
||||
}
|
||||
@@ -0,0 +1,525 @@
|
||||
package fileedit
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// diskStage is a stage on a disk of total bytes whose free space is what the
|
||||
// files in it leave of free: statfs sees parts land, as a real disk would.
|
||||
func diskStage(t *testing.T, free, total uint64, minFree float64) *Stage {
|
||||
t.Helper()
|
||||
s := &Stage{Dir: filepath.Join(t.TempDir(), "stage"), MinFree: minFree}
|
||||
prev := statfs
|
||||
statfs = func(string) (uint64, uint64, error) {
|
||||
var used uint64
|
||||
des, _ := os.ReadDir(s.Dir)
|
||||
for _, de := range des {
|
||||
if info, err := de.Info(); err == nil {
|
||||
used += uint64(info.Size())
|
||||
}
|
||||
}
|
||||
return free - used, total, nil
|
||||
}
|
||||
t.Cleanup(func() { statfs = prev })
|
||||
return s
|
||||
}
|
||||
|
||||
func appendString(s *Stage, user, server, id string, offset int64, part string) (Session, error) {
|
||||
return s.Append(user, server, id, offset, strings.NewReader(part), int64(len(part)))
|
||||
}
|
||||
|
||||
func readStaged(t *testing.T, s *Stage, id, token string) string {
|
||||
t.Helper()
|
||||
f, size, err := s.Open(id, token)
|
||||
if err != nil {
|
||||
t.Fatalf("Open: %v", err)
|
||||
}
|
||||
defer f.Close()
|
||||
b, err := io.ReadAll(f)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if size != int64(len(b)) {
|
||||
t.Fatalf("Open said %d bytes and served %d", size, len(b))
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
// TestSessionArrivesInParts: parts land in order, Seal hands the Job a token for
|
||||
// exactly those bytes, and Served deletes them.
|
||||
func TestSessionArrivesInParts(t *testing.T) {
|
||||
s := roomyStage(t)
|
||||
const whole = "PK\x03\x04 first part, second part"
|
||||
sess, err := s.Begin("u1", "survival", "plugins/big.jar", int64(len(whole)))
|
||||
if err != nil {
|
||||
t.Fatalf("Begin: %v", err)
|
||||
}
|
||||
if !hexID.MatchString(sess.ID) || sess != (Session{ID: sess.ID, Path: "plugins/big.jar", Size: int64(len(whole))}) {
|
||||
t.Fatalf("Begin = %+v", sess)
|
||||
}
|
||||
got, err := appendString(s, "u1", "survival", sess.ID, 0, whole[:16])
|
||||
if err != nil || got.Received != 16 || got.Size != int64(len(whole)) {
|
||||
t.Fatalf("first part: %+v, %v", got, err)
|
||||
}
|
||||
if at, err := s.Status("u1", "survival", sess.ID); err != nil || at.Received != 16 || at.Path != "plugins/big.jar" {
|
||||
t.Fatalf("Status = %+v, %v", at, err)
|
||||
}
|
||||
if got, err = appendString(s, "u1", "survival", sess.ID, 16, whole[16:]); err != nil || got.Received != int64(len(whole)) {
|
||||
t.Fatalf("second part: %+v, %v", got, err)
|
||||
}
|
||||
|
||||
st, err := s.Seal("u1", "survival", sess.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("Seal: %v", err)
|
||||
}
|
||||
if st.ID != sess.ID || !hexToken.MatchString(st.Token) || st.Size != int64(len(whole)) || st.SHA256 != digest([]byte(whole)) {
|
||||
t.Fatalf("Seal = %+v, want the digest of %q", st, whole)
|
||||
}
|
||||
if body := readStaged(t, s, st.ID, st.Token); body != whole {
|
||||
t.Fatalf("served %q, want %q", body, whole)
|
||||
}
|
||||
if _, _, err := s.Open(st.ID, st.Token); !errors.Is(err, ErrNotStaged) {
|
||||
t.Fatalf("second Open with the same token: err = %v, want ErrNotStaged", err)
|
||||
}
|
||||
|
||||
s.Served(st.ID)
|
||||
if names := stagedNames(t, s); len(names) != 0 {
|
||||
t.Fatalf("after Served: %v", names)
|
||||
}
|
||||
if _, err := s.Status("u1", "survival", sess.ID); !errors.Is(err, ErrNotStaged) {
|
||||
t.Fatalf("Status after Served: err = %v, want ErrNotStaged", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A session answers only the user and the server it was begun for.
|
||||
func TestSessionAnswersItsOwnerOnly(t *testing.T) {
|
||||
s := roomyStage(t)
|
||||
sess, err := s.Begin("u1", "survival", "a.zip", 4)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for name, who := range map[string][2]string{
|
||||
"another user": {"u2", "survival"},
|
||||
"another server": {"u1", "creative"},
|
||||
} {
|
||||
if _, err := s.Status(who[0], who[1], sess.ID); !errors.Is(err, ErrNotStaged) {
|
||||
t.Errorf("%s: Status err = %v", name, err)
|
||||
}
|
||||
if _, err := appendString(s, who[0], who[1], sess.ID, 0, "abcd"); !errors.Is(err, ErrNotStaged) {
|
||||
t.Errorf("%s: Append err = %v", name, err)
|
||||
}
|
||||
if _, err := s.Seal(who[0], who[1], sess.ID); !errors.Is(err, ErrNotStaged) {
|
||||
t.Errorf("%s: Seal err = %v", name, err)
|
||||
}
|
||||
if err := s.Drop(who[0], who[1], sess.ID); !errors.Is(err, ErrNotStaged) {
|
||||
t.Errorf("%s: Drop err = %v", name, err)
|
||||
}
|
||||
}
|
||||
if _, err := s.Status("u1", "survival", strings.Repeat("0", 32)); !errors.Is(err, ErrNotStaged) {
|
||||
t.Errorf("unknown id: err = %v", err)
|
||||
}
|
||||
if at, err := s.Status("u1", "survival", sess.ID); err != nil || at.Received != 0 {
|
||||
t.Fatalf("the owner's session after the others tried: %+v, %v", at, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSessionRefusesAPartThatDoesNotFit(t *testing.T) {
|
||||
s := roomyStage(t)
|
||||
sess, err := s.Begin("u1", "survival", "a.zip", 6)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := appendString(s, "u1", "survival", sess.ID, 0, "abc"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
var off *OffsetError
|
||||
for _, offset := range []int64{0, 2, 4} {
|
||||
at, err := appendString(s, "u1", "survival", sess.ID, offset, "d")
|
||||
if !errors.As(err, &off) || off.Received != 3 || at.Received != 3 {
|
||||
t.Fatalf("offset %d: %+v, err = %v; want an OffsetError at 3", offset, at, err)
|
||||
}
|
||||
}
|
||||
if _, err := appendString(s, "u1", "survival", sess.ID, 3, "defg"); !errors.Is(err, ErrPartTooLarge) {
|
||||
t.Fatalf("past the declared size: err = %v, want ErrPartTooLarge", err)
|
||||
}
|
||||
if _, err := s.Append("u1", "survival", sess.ID, 3, strings.NewReader(""), -1); !errors.Is(err, ErrPartTooLarge) {
|
||||
t.Fatalf("negative length: err = %v, want ErrPartTooLarge", err)
|
||||
}
|
||||
if at, err := appendString(s, "u1", "survival", sess.ID, 3, "def"); err != nil || at.Received != 6 {
|
||||
t.Fatalf("the part that fits exactly: %+v, %v", at, err)
|
||||
}
|
||||
|
||||
big, err := s.Begin("u1", "survival", "b.zip", PartBytes+2)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := s.Append("u1", "survival", big.ID, 0, strings.NewReader(""), PartBytes+1); !errors.Is(err, ErrPartTooLarge) {
|
||||
t.Fatalf("a part over PartBytes: err = %v, want ErrPartTooLarge", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A part that breaks or runs long leaves the session as it was: the file is cut
|
||||
// back and the digest forgets it, so the resent part makes the right file.
|
||||
func TestSessionRollsBackAFailedPart(t *testing.T) {
|
||||
for name, tc := range map[string]struct {
|
||||
body io.Reader
|
||||
short bool
|
||||
}{
|
||||
"breaks": {io.MultiReader(strings.NewReader("XY"), errReader{io.ErrUnexpectedEOF}), true},
|
||||
"ends": {strings.NewReader("XY"), true},
|
||||
"runs long": {strings.NewReader("XYZWV"), false},
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
s := roomyStage(t)
|
||||
sess, err := s.Begin("u1", "survival", "a.zip", 7)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := appendString(s, "u1", "survival", sess.ID, 0, "abc"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
at, err := s.Append("u1", "survival", sess.ID, 3, tc.body, 4)
|
||||
if err == nil || errors.Is(err, ErrShortUpload) != tc.short || at.Received != 3 {
|
||||
t.Fatalf("%+v, err = %v; want a failure at 3 (short = %v)", at, err, tc.short)
|
||||
}
|
||||
info, err := os.Stat(filepath.Join(s.Dir, stagedNames(t, s)[0]))
|
||||
if err != nil || info.Size() != 3 {
|
||||
t.Fatalf("staged file is %v bytes (%v), want it cut back to 3", info.Size(), err)
|
||||
}
|
||||
if _, err := appendString(s, "u1", "survival", sess.ID, 3, "defg"); err != nil {
|
||||
t.Fatalf("resent part: %v", err)
|
||||
}
|
||||
st, err := s.Seal("u1", "survival", sess.ID)
|
||||
if err != nil || st.SHA256 != digest([]byte("abcdefg")) {
|
||||
t.Fatalf("Seal = %+v, %v; want the digest of abcdefg", st, err)
|
||||
}
|
||||
if body := readStaged(t, s, st.ID, st.Token); body != "abcdefg" {
|
||||
t.Fatalf("served %q", body)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// While a part is arriving nothing else may touch the session, and it is never
|
||||
// idle.
|
||||
func TestSessionIsBusyWhileAPartArrives(t *testing.T) {
|
||||
s := roomyStage(t)
|
||||
now := time.Date(2026, 9, 28, 12, 0, 0, 0, time.UTC)
|
||||
s.Now = func() time.Time { return now }
|
||||
sess, err := s.Begin("u1", "survival", "a.zip", 4)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
pr, pw := io.Pipe()
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
_, err := s.Append("u1", "survival", sess.ID, 0, pr, 4)
|
||||
done <- err
|
||||
}()
|
||||
// The write returns once Append is copying, which is after it marked busy.
|
||||
if _, err := pw.Write([]byte("ab")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := appendString(s, "u1", "survival", sess.ID, 0, "abcd"); !errors.Is(err, ErrUploadBusy) {
|
||||
t.Errorf("a second part: err = %v, want ErrUploadBusy", err)
|
||||
}
|
||||
if _, err := s.Seal("u1", "survival", sess.ID); !errors.Is(err, ErrUploadBusy) {
|
||||
t.Errorf("Seal: err = %v, want ErrUploadBusy", err)
|
||||
}
|
||||
if err := s.Drop("u1", "survival", sess.ID); !errors.Is(err, ErrUploadBusy) {
|
||||
t.Errorf("Drop: err = %v, want ErrUploadBusy", err)
|
||||
}
|
||||
now = now.Add(SessionIdle + time.Hour)
|
||||
if n := s.Expire(); n != 0 {
|
||||
t.Errorf("Expire dropped %d sessions with a part arriving", n)
|
||||
}
|
||||
if _, err := pw.Write([]byte("cd")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
pw.Close()
|
||||
if err := <-done; err != nil {
|
||||
t.Fatalf("the part in flight: %v", err)
|
||||
}
|
||||
if _, err := s.Seal("u1", "survival", sess.ID); err != nil {
|
||||
t.Fatalf("Seal once the part is in: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Each Seal arms one fetch with a fresh token, so a Job that failed can be
|
||||
// started again on the same bytes.
|
||||
func TestSessionSealArmsOneFetch(t *testing.T) {
|
||||
s := roomyStage(t)
|
||||
sess, err := s.Begin("u1", "survival", "a.zip", 4)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := appendString(s, "u1", "survival", sess.ID, 0, "abc"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := s.Seal("u1", "survival", sess.ID); !errors.Is(err, ErrUploadIncomplete) {
|
||||
t.Fatalf("Seal at 3 of 4: err = %v, want ErrUploadIncomplete", err)
|
||||
}
|
||||
if _, _, err := s.Open(sess.ID, ""); !errors.Is(err, ErrNotStaged) {
|
||||
t.Fatalf("Open before any Seal: err = %v, want ErrNotStaged", err)
|
||||
}
|
||||
if _, err := appendString(s, "u1", "survival", sess.ID, 3, "d"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
first, err := s.Seal("u1", "survival", sess.ID)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
second, err := s.Seal("u1", "survival", sess.ID)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if first.Token == second.Token || first.SHA256 != second.SHA256 {
|
||||
t.Fatalf("two Seals: %+v and %+v; want fresh tokens for the same bytes", first, second)
|
||||
}
|
||||
if _, _, err := s.Open(sess.ID, first.Token); !errors.Is(err, ErrNotStaged) {
|
||||
t.Fatalf("the replaced token: err = %v, want ErrNotStaged", err)
|
||||
}
|
||||
if _, _, err := s.Open(sess.ID, strings.Repeat("0", 64)); !errors.Is(err, ErrNotStaged) {
|
||||
t.Fatalf("a wrong token: err = %v, want ErrNotStaged", err)
|
||||
}
|
||||
// Neither wrong token spent the armed one.
|
||||
if body := readStaged(t, s, sess.ID, second.Token); body != "abcd" {
|
||||
t.Fatalf("served %q", body)
|
||||
}
|
||||
if _, _, err := s.Open(sess.ID, second.Token); !errors.Is(err, ErrNotStaged) {
|
||||
t.Fatalf("the spent token: err = %v, want ErrNotStaged", err)
|
||||
}
|
||||
|
||||
// Opened but not served whole: the Job broke midway, and a new Seal serves
|
||||
// the same bytes again.
|
||||
third, err := s.Seal("u1", "survival", sess.ID)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if body := readStaged(t, s, sess.ID, third.Token); body != "abcd" {
|
||||
t.Fatalf("served %q after a new Seal", body)
|
||||
}
|
||||
}
|
||||
|
||||
// Served deletes only sessions: an upload staged by Put belongs to the release
|
||||
// func Put returned.
|
||||
func TestServedLeavesPutAlone(t *testing.T) {
|
||||
s := roomyStage(t)
|
||||
st, release, err := s.Put(strings.NewReader("abc"), 3)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer release()
|
||||
s.Served(st.ID)
|
||||
if body := readStaged(t, s, st.ID, st.Token); body != "abc" {
|
||||
t.Fatalf("served %q", body)
|
||||
}
|
||||
}
|
||||
|
||||
// A session reserves its whole size when it begins, and gives back what it has
|
||||
// not yet received when it is dropped or expires.
|
||||
func TestSessionReservesItsSize(t *testing.T) {
|
||||
t.Run("begin reserves the whole size", func(t *testing.T) {
|
||||
s := diskStage(t, 1000, 1200, 0.5) // room for 400
|
||||
if _, err := s.Begin("u1", "survival", "a.zip", 300); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := s.Begin("u2", "survival", "b.zip", 101); !errors.Is(err, ErrStageFull) {
|
||||
t.Fatalf("101 bytes beside a 300-byte session: err = %v, want ErrStageFull", err)
|
||||
}
|
||||
if _, err := s.Begin("u2", "survival", "b.zip", 100); err != nil {
|
||||
t.Fatalf("100 bytes beside a 300-byte session: %v", err)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("a part moves its room from the reservation to the disk", func(t *testing.T) {
|
||||
s := diskStage(t, 1000, 1200, 0.5)
|
||||
sess, err := s.Begin("u1", "survival", "a.zip", 300)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := appendString(s, "u1", "survival", sess.ID, 0, strings.Repeat("x", 200)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := s.Begin("u2", "survival", "b.zip", 101); !errors.Is(err, ErrStageFull) {
|
||||
t.Fatalf("after a part landed: err = %v, want ErrStageFull", err)
|
||||
}
|
||||
if _, err := s.Begin("u2", "survival", "b.zip", 100); err != nil {
|
||||
t.Fatalf("after a part landed: %v", err)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("drop gives it all back", func(t *testing.T) {
|
||||
s := diskStage(t, 1000, 1200, 0.5)
|
||||
sess, err := s.Begin("u1", "survival", "a.zip", 300)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := appendString(s, "u1", "survival", sess.ID, 0, strings.Repeat("x", 200)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.Drop("u1", "survival", sess.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if names := stagedNames(t, s); len(names) != 0 {
|
||||
t.Fatalf("after Drop: %v", names)
|
||||
}
|
||||
if _, err := s.Begin("u2", "survival", "b.zip", 400); err != nil {
|
||||
t.Fatalf("after Drop: %v", err)
|
||||
}
|
||||
if _, err := s.Status("u1", "survival", sess.ID); !errors.Is(err, ErrNotStaged) {
|
||||
t.Fatalf("Status after Drop: err = %v", err)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("expiry gives it all back", func(t *testing.T) {
|
||||
s := diskStage(t, 1000, 1200, 0.5)
|
||||
now := time.Date(2026, 9, 28, 12, 0, 0, 0, time.UTC)
|
||||
s.Now = func() time.Time { return now }
|
||||
sess, err := s.Begin("u1", "survival", "a.zip", 300)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := appendString(s, "u1", "survival", sess.ID, 0, strings.Repeat("x", 200)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
now = now.Add(SessionIdle + time.Nanosecond)
|
||||
if n := s.Expire(); n != 1 {
|
||||
t.Fatalf("Expire dropped %d, want 1", n)
|
||||
}
|
||||
if _, err := s.Begin("u2", "survival", "b.zip", 400); err != nil {
|
||||
t.Fatalf("after Expire: %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestSessionsPerUserAreBounded(t *testing.T) {
|
||||
s := diskStage(t, 1000, 1200, 0.5)
|
||||
var ids []string
|
||||
for i := range MaxSessionsPerUser {
|
||||
sess, err := s.Begin("u1", "survival", "a.zip", 10)
|
||||
if err != nil {
|
||||
t.Fatalf("session %d: %v", i+1, err)
|
||||
}
|
||||
ids = append(ids, sess.ID)
|
||||
}
|
||||
if _, err := s.Begin("u1", "creative", "a.zip", 10); !errors.Is(err, ErrTooManySessions) {
|
||||
t.Fatalf("one more on another server: err = %v, want ErrTooManySessions", err)
|
||||
}
|
||||
// The refused Begin kept neither a file nor its reservation: room for 400
|
||||
// less the four sessions' 40.
|
||||
if names := stagedNames(t, s); len(names) != MaxSessionsPerUser {
|
||||
t.Fatalf("staged %d files, want %d", len(names), MaxSessionsPerUser)
|
||||
}
|
||||
if _, err := s.Begin("u2", "survival", "b.zip", 360); err != nil {
|
||||
t.Fatalf("another user: %v", err)
|
||||
}
|
||||
if err := s.Drop("u1", "survival", ids[0]); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := s.Begin("u1", "survival", "a.zip", 10); err != nil {
|
||||
t.Fatalf("after dropping one: %v", err)
|
||||
}
|
||||
// With room reserved, a negative size would wrap the reservation sum round
|
||||
// to a small number and pass the room check.
|
||||
if _, err := s.Begin("u3", "survival", "a.zip", -1); err == nil || errors.Is(err, ErrStageFull) {
|
||||
t.Fatalf("a negative size: err = %v, want it refused for being negative", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Expire drops what has sat untouched for longer than SessionIdle, sealed or
|
||||
// not, and a part keeps a session alive.
|
||||
func TestSessionExpiry(t *testing.T) {
|
||||
s := roomyStage(t)
|
||||
start := time.Date(2026, 9, 28, 12, 0, 0, 0, time.UTC)
|
||||
now := start
|
||||
s.Now = func() time.Time { return now }
|
||||
idle, err := s.Begin("u1", "survival", "idle.zip", 2)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
sealed, err := s.Begin("u1", "survival", "sealed.zip", 1)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := appendString(s, "u1", "survival", sealed.ID, 0, "x"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := s.Seal("u1", "survival", sealed.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
active, err := s.Begin("u1", "survival", "active.zip", 2)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
now = start.Add(time.Hour)
|
||||
if _, err := appendString(s, "u1", "survival", active.ID, 0, "a"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
now = start.Add(SessionIdle)
|
||||
if n := s.Expire(); n != 0 {
|
||||
t.Fatalf("at exactly SessionIdle Expire dropped %d", n)
|
||||
}
|
||||
now = start.Add(SessionIdle + time.Minute)
|
||||
if n := s.Expire(); n != 2 {
|
||||
t.Fatalf("Expire dropped %d, want the idle and the sealed one", n)
|
||||
}
|
||||
for _, id := range []string{idle.ID, sealed.ID} {
|
||||
if _, err := s.Status("u1", "survival", id); !errors.Is(err, ErrNotStaged) {
|
||||
t.Errorf("%s survived Expire: %v", id, err)
|
||||
}
|
||||
}
|
||||
if at, err := s.Status("u1", "survival", active.ID); err != nil || at.Received != 1 {
|
||||
t.Fatalf("the session a part touched: %+v, %v", at, err)
|
||||
}
|
||||
if names := stagedNames(t, s); len(names) != 1 {
|
||||
t.Fatalf("files left: %v, want the active session's", names)
|
||||
}
|
||||
|
||||
// A Seal, and the Job's Open, each start the idle clock again: a Job begun
|
||||
// on an upload that sat for hours still finds it there.
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
// sealAt and openAt are how long after the last part the Seal and the
|
||||
// Job's Open come; a negative openAt is no Open.
|
||||
sealAt, openAt time.Duration
|
||||
}{
|
||||
{"seal", 4 * time.Hour, -1},
|
||||
{"open", 0, 4 * time.Hour},
|
||||
} {
|
||||
begun := now
|
||||
sess, err := s.Begin("u1", "survival", tc.name+".zip", 1)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := appendString(s, "u1", "survival", sess.ID, 0, "x"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
now = begun.Add(tc.sealAt)
|
||||
st, err := s.Seal("u1", "survival", sess.ID)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if tc.openAt >= 0 {
|
||||
now = begun.Add(tc.openAt)
|
||||
readStaged(t, s, sess.ID, st.Token)
|
||||
}
|
||||
now = begun.Add(SessionIdle + time.Minute)
|
||||
s.Expire()
|
||||
if _, err := s.Status("u1", "survival", sess.ID); err != nil {
|
||||
t.Errorf("%s: a session touched 4h after its last part expired 6h after it: %v", tc.name, err)
|
||||
}
|
||||
if err := s.Drop("u1", "survival", sess.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
+38
-20
@@ -12,6 +12,7 @@ import (
|
||||
"os"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Stage holds uploads between the request that brought them and the Job that
|
||||
@@ -25,7 +26,9 @@ import (
|
||||
// Job carries in its environment. Only a digest of the token is kept, compared in
|
||||
// constant time, and the first successful Open spends it: the Job never retries,
|
||||
// so a second Open could only be someone else. The release func Put returns
|
||||
// deletes the file once the Job has answered, whatever it answered.
|
||||
// deletes the file once the Job has answered, whatever it answered. A file too
|
||||
// big for one request arrives in parts instead (session.go) and is fetched the
|
||||
// same way.
|
||||
//
|
||||
// Nothing here outlives the process: the index is in memory, so Sweep empties
|
||||
// Dir at startup of whatever a previous process left behind.
|
||||
@@ -38,8 +41,12 @@ type Stage struct {
|
||||
// the submission store shares.
|
||||
MinFree float64
|
||||
|
||||
// Now is the clock sessions are aged by (time.Now when nil).
|
||||
Now func() time.Time
|
||||
|
||||
mu sync.Mutex
|
||||
items map[string]*stagedFile
|
||||
sessions map[string]*session
|
||||
reserved int64
|
||||
}
|
||||
|
||||
@@ -172,23 +179,27 @@ func stageFailure(readErr, copyErr, closeErr error, n, size int64) error {
|
||||
|
||||
// newHandle mints the id and token for a staged upload whose bytes h hashed.
|
||||
func newHandle(h hash.Hash, size int64) (Staged, [sha256.Size]byte, error) {
|
||||
var id [16]byte
|
||||
var token [32]byte
|
||||
if _, err := rand.Read(id[:]); err != nil {
|
||||
id, err := randomHex(16)
|
||||
if err != nil {
|
||||
return Staged{}, [sha256.Size]byte{}, fmt.Errorf("fileedit: generate an upload id: %w", err)
|
||||
}
|
||||
if _, err := rand.Read(token[:]); err != nil {
|
||||
token, err := randomHex(32)
|
||||
if err != nil {
|
||||
return Staged{}, [sha256.Size]byte{}, fmt.Errorf("fileedit: generate an upload token: %w", err)
|
||||
}
|
||||
st := Staged{
|
||||
ID: hex.EncodeToString(id[:]),
|
||||
Token: hex.EncodeToString(token[:]),
|
||||
SHA256: hex.EncodeToString(h.Sum(nil)),
|
||||
Size: size,
|
||||
}
|
||||
st := Staged{ID: id, Token: token, SHA256: hex.EncodeToString(h.Sum(nil)), Size: size}
|
||||
return st, sha256.Sum256([]byte(st.Token)), nil
|
||||
}
|
||||
|
||||
// randomHex is n random bytes in hex.
|
||||
func randomHex(n int) (string, error) {
|
||||
b := make([]byte, n)
|
||||
if _, err := rand.Read(b); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return hex.EncodeToString(b), nil
|
||||
}
|
||||
|
||||
// reserve admits an upload of size bytes if the disk keeps MinFree free after it
|
||||
// and after every upload still being written. Those have not reached the disk
|
||||
// yet, so statfs alone would let two of them through on room for one.
|
||||
@@ -219,23 +230,30 @@ func (s *Stage) unreserve(size int64) {
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
// Open spends a staged upload's token and returns its file and size. Any
|
||||
// mismatch is ErrNotStaged.
|
||||
// Open spends a staged upload's token, or a sealed session's (Seal), and returns
|
||||
// its file and size. Any mismatch is ErrNotStaged.
|
||||
func (s *Stage) Open(id, token string) (*os.File, int64, error) {
|
||||
sum := sha256.Sum256([]byte(token))
|
||||
s.mu.Lock()
|
||||
it, ok := s.items[id]
|
||||
if !ok || it.used || subtle.ConstantTimeCompare(sum[:], it.tokenHash[:]) != 1 {
|
||||
s.mu.Unlock()
|
||||
return nil, 0, ErrNotStaged
|
||||
name, size, found, err := s.openSession(id, sum)
|
||||
if !found {
|
||||
it, ok := s.items[id]
|
||||
if !ok || it.used || subtle.ConstantTimeCompare(sum[:], it.tokenHash[:]) != 1 {
|
||||
err = ErrNotStaged
|
||||
} else {
|
||||
it.used = true
|
||||
name, size = it.path, it.size
|
||||
}
|
||||
}
|
||||
it.used = true
|
||||
s.mu.Unlock()
|
||||
f, err := os.Open(it.path)
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
f, err := os.Open(name)
|
||||
if err != nil {
|
||||
return nil, 0, fmt.Errorf("fileedit: open the staged upload: %w", err)
|
||||
}
|
||||
return f, it.size, nil
|
||||
return f, size, nil
|
||||
}
|
||||
|
||||
// bodyReader remembers the body's own read error, so Put can tell a client that
|
||||
|
||||
@@ -0,0 +1,536 @@
|
||||
package fileedit
|
||||
|
||||
import (
|
||||
"archive/zip"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"io/fs"
|
||||
"math"
|
||||
"os"
|
||||
"path"
|
||||
"sort"
|
||||
"strings"
|
||||
"unicode/utf8"
|
||||
|
||||
"golang.org/x/text/encoding/simplifiedchinese"
|
||||
)
|
||||
|
||||
// The codes an unzip refuses an archive with. Each names what is wrong with the
|
||||
// ARCHIVE, so the panel can say "this zip is broken" rather than "your path is
|
||||
// wrong"; Result.Entry names the entry.
|
||||
const (
|
||||
// CodeArchiveInvalid is a file that is not a zip, a damaged one, one whose
|
||||
// entry bytes disagree with their header (size or CRC), or one with nothing in it.
|
||||
CodeArchiveInvalid = "archive_invalid"
|
||||
// CodeArchiveUnsafe is an entry naming a path outside the folder it is
|
||||
// extracted into ("../", "/etc/x", "C:\x"), or a device, pipe or socket.
|
||||
CodeArchiveUnsafe = "archive_unsafe"
|
||||
// CodeArchiveSymlink is an entry that is a symbolic link. A link extracted into
|
||||
// the world could point anywhere, and every later op would have to reason about
|
||||
// it, so an archive carrying one is refused whole.
|
||||
CodeArchiveSymlink = "archive_symlink"
|
||||
// CodeTypeConflict is an archive with a file where the server has a folder, a
|
||||
// folder where it has a file, or anything where it has a link. Overwrite never
|
||||
// resolves it: replacing a folder with a file would delete the folder.
|
||||
CodeTypeConflict = "type_conflict"
|
||||
)
|
||||
|
||||
// MaxConflicts bounds Result.Conflicts, keeping the result line bounded when an
|
||||
// archive would replace a whole world; ConflictCount still says how many there
|
||||
// are.
|
||||
const MaxConflicts = 200
|
||||
|
||||
// unzipEntryOverhead is what the space check adds per entry for the inode and
|
||||
// directory block it takes beyond its bytes.
|
||||
const unzipEntryOverhead = 4096
|
||||
|
||||
// unzipTempPrefix names the working folder an unzip extracts into. It sits in
|
||||
// the destination folder, so every move out of it is a rename on one volume.
|
||||
const unzipTempPrefix = ".felis-unzip-"
|
||||
|
||||
// unzip extracts the .zip at name into the folder holding it.
|
||||
//
|
||||
// It is all or nothing. Every check that can refuse the archive runs before a
|
||||
// byte is written: entry names, entry types, what is already on the server, and
|
||||
// the room on the volume. The entries are then extracted into a working folder
|
||||
// beside the destination, and only once every one of them has been written and
|
||||
// verified are they renamed into place. A failure at any point before that
|
||||
// leaves the destination exactly as it was. The renames themselves are journaled
|
||||
// and undone in reverse if one fails, so a replaced file comes back. Only a Job
|
||||
// killed in the middle of the renames — a few milliseconds for thousands of
|
||||
// files — can leave the archive half applied.
|
||||
//
|
||||
// An existing file the archive would replace is a conflict: without overwrite the
|
||||
// unzip lists them (CodeExists, Conflicts) and changes nothing; with it they are
|
||||
// replaced. Folders merge. A file where the server has a folder, or the reverse,
|
||||
// is CodeTypeConflict whatever overwrite says.
|
||||
//
|
||||
// The only size bound is the volume. Each entry's declared size is summed and
|
||||
// checked against the free space up front, and archive/zip itself refuses an
|
||||
// entry whose bytes run past its declared size or fail its CRC, so an archive
|
||||
// that lies about its sizes (a zip bomb) stops at the first lying entry and
|
||||
// nothing it wrote survives.
|
||||
func unzip(r *os.Root, rootPath, name string, overwrite bool, progress func(done, total int64)) Result {
|
||||
name = path.Clean(name)
|
||||
if !strings.EqualFold(path.Ext(name), ".zip") {
|
||||
return Result{Code: CodeBadPath, Error: fmt.Sprintf("%s is not a .zip archive", name)}
|
||||
}
|
||||
f, err := r.Open(name)
|
||||
if err != nil {
|
||||
return failure(err, name)
|
||||
}
|
||||
defer f.Close()
|
||||
info, err := f.Stat()
|
||||
if err != nil {
|
||||
return failure(err, name)
|
||||
}
|
||||
if !info.Mode().IsRegular() {
|
||||
return Result{Code: CodeBadPath, Error: fmt.Sprintf("%s is not a file", name)}
|
||||
}
|
||||
// ErrInsecurePath comes back WITH a usable reader, and only under
|
||||
// GODEBUG=zipinsecurepath=0; planUnzip does that check itself, for every
|
||||
// entry, whatever the setting.
|
||||
zr, err := zip.NewReader(f, info.Size())
|
||||
if err != nil && !errors.Is(err, zip.ErrInsecurePath) {
|
||||
return Result{Code: CodeArchiveInvalid, Error: fmt.Sprintf("%s is not a readable zip archive: %v", name, err)}
|
||||
}
|
||||
p, res := planUnzip(zr.File)
|
||||
if res.Code != "" {
|
||||
return res
|
||||
}
|
||||
|
||||
dest := path.Dir(name)
|
||||
present, replaced, res := checkTargets(r, dest, p)
|
||||
if res.Code != "" {
|
||||
return res
|
||||
}
|
||||
if len(replaced) > 0 && !overwrite {
|
||||
list := make([]string, 0, len(replaced))
|
||||
for n := range replaced {
|
||||
list = append(list, path.Join(dest, n))
|
||||
}
|
||||
sort.Strings(list)
|
||||
count := len(list)
|
||||
if count > MaxConflicts {
|
||||
list = list[:MaxConflicts]
|
||||
}
|
||||
return Result{Code: CodeExists, Conflicts: list, ConflictCount: count, Error: fmt.Sprintf(
|
||||
"%d files in the archive already exist on the server; extract again with overwrite to replace them", count)}
|
||||
}
|
||||
|
||||
// A working folder left by an unzip that was killed holds only a copy, and
|
||||
// clearing it first gives its room back to the check below.
|
||||
sweepUnzipTemps(r, dest)
|
||||
need := p.bytes + int64(len(p.files)+len(p.dirs))*unzipEntryOverhead
|
||||
if avail, _, err := statfs(rootPath); err == nil && uint64(need) > avail {
|
||||
free := int64(math.MaxInt64)
|
||||
if avail < math.MaxInt64 {
|
||||
free = int64(avail)
|
||||
}
|
||||
return Result{Code: CodeNoSpace, Need: need, Avail: free, Error: fmt.Sprintf(
|
||||
"extracting %s needs %d bytes and the server's volume has %d free; nothing was changed", name, need, free)}
|
||||
}
|
||||
|
||||
var suffix [6]byte
|
||||
if _, err := rand.Read(suffix[:]); err != nil {
|
||||
return Result{Code: CodeBadPath, Error: fmt.Sprintf("generate a temporary name: %v", err)}
|
||||
}
|
||||
tmp := path.Join(dest, unzipTempPrefix+hex.EncodeToString(suffix[:]))
|
||||
staged, old := path.Join(tmp, "new"), path.Join(tmp, "old")
|
||||
for _, d := range []string{tmp, staged, old} {
|
||||
if err := r.Mkdir(d, 0o700); err != nil {
|
||||
_ = r.RemoveAll(tmp)
|
||||
return unzipWriteFailure(err, dest)
|
||||
}
|
||||
}
|
||||
// After a success tmp holds only the files the archive replaced; after a
|
||||
// failure, everything the archive wrote. Either way it goes.
|
||||
defer func() { _ = r.RemoveAll(tmp) }()
|
||||
|
||||
if res := extractAll(r, staged, p, progress); res.Code != "" {
|
||||
return res
|
||||
}
|
||||
if res := placeAll(r, dest, staged, old, p, present, replaced); res.Code != "" {
|
||||
return res
|
||||
}
|
||||
return Result{Files: len(p.files), Bytes: p.bytes}
|
||||
}
|
||||
|
||||
// unzipPlan is an archive's entries once every one has passed planUnzip. Names
|
||||
// are cleaned, slash-separated and relative to the destination folder.
|
||||
type unzipPlan struct {
|
||||
files []zipFile
|
||||
// dirs is every folder the archive makes, named in it or implied by a file
|
||||
// inside it, sorted so a folder comes before everything in it.
|
||||
dirs []string
|
||||
isDir map[string]bool
|
||||
bytes int64 // the declared size of every file, summed
|
||||
}
|
||||
|
||||
type zipFile struct {
|
||||
f *zip.File
|
||||
name string
|
||||
mode fs.FileMode
|
||||
}
|
||||
|
||||
// planUnzip checks every entry's name and type and works out what the archive
|
||||
// makes. Nothing about the server is consulted yet.
|
||||
func planUnzip(entries []*zip.File) (unzipPlan, Result) {
|
||||
p := unzipPlan{isDir: map[string]bool{}}
|
||||
byName := map[string]bool{}
|
||||
for _, f := range entries {
|
||||
raw := entryName(f)
|
||||
name, ok := cleanEntry(raw)
|
||||
if !ok {
|
||||
return p, Result{Code: CodeArchiveUnsafe, Entry: raw, Error: fmt.Sprintf(
|
||||
"%s leads outside the folder it would be extracted into", raw)}
|
||||
}
|
||||
// macOS's Finder adds __MACOSX/ to every zip it makes: resource forks that
|
||||
// mean nothing on the server.
|
||||
if name == "__MACOSX" || strings.HasPrefix(name, "__MACOSX/") {
|
||||
continue
|
||||
}
|
||||
mode := f.Mode()
|
||||
isDir := mode.IsDir() || strings.HasSuffix(raw, "/")
|
||||
switch {
|
||||
case mode&fs.ModeSymlink != 0:
|
||||
return p, Result{Code: CodeArchiveSymlink, Entry: raw, Error: fmt.Sprintf(
|
||||
"%s is a symbolic link; an archive containing links is not extracted", raw)}
|
||||
case isDir:
|
||||
if name != "." {
|
||||
p.isDir[name] = true
|
||||
}
|
||||
continue
|
||||
case !mode.IsRegular():
|
||||
return p, Result{Code: CodeArchiveUnsafe, Entry: raw, Error: fmt.Sprintf(
|
||||
"%s is not a regular file", raw)}
|
||||
case name == ".":
|
||||
return p, Result{Code: CodeArchiveUnsafe, Entry: raw, Error: fmt.Sprintf(
|
||||
"%s names the destination folder itself", raw)}
|
||||
case byName[name]:
|
||||
return p, Result{Code: CodeArchiveInvalid, Entry: raw, Error: fmt.Sprintf(
|
||||
"%s appears in the archive twice", raw)}
|
||||
case f.UncompressedSize64 > uint64(math.MaxInt64-p.bytes):
|
||||
return p, Result{Code: CodeArchiveInvalid, Entry: raw, Error: fmt.Sprintf(
|
||||
"%s declares an impossible size", raw)}
|
||||
}
|
||||
byName[name] = true
|
||||
p.bytes += int64(f.UncompressedSize64)
|
||||
// Anything the archive marks executable (a start.sh) stays executable;
|
||||
// every other permission is the server's usual.
|
||||
perm := fs.FileMode(0o644)
|
||||
if mode.Perm()&0o111 != 0 {
|
||||
perm = 0o755
|
||||
}
|
||||
p.files = append(p.files, zipFile{f: f, name: name, mode: perm})
|
||||
}
|
||||
for _, zf := range p.files {
|
||||
// cleanEntry already refused a rooted name; stopping at "/" as well keeps
|
||||
// this loop finite should that check ever move.
|
||||
for d := path.Dir(zf.name); d != "." && d != "/"; d = path.Dir(d) {
|
||||
p.isDir[d] = true
|
||||
}
|
||||
}
|
||||
for _, zf := range p.files {
|
||||
if p.isDir[zf.name] {
|
||||
return p, Result{Code: CodeArchiveInvalid, Entry: zf.name, Error: fmt.Sprintf(
|
||||
"%s is both a file and a folder in the archive", zf.name)}
|
||||
}
|
||||
}
|
||||
if len(p.files) == 0 && len(p.isDir) == 0 {
|
||||
return p, Result{Code: CodeArchiveInvalid, Error: "the archive has nothing to extract"}
|
||||
}
|
||||
for d := range p.isDir {
|
||||
p.dirs = append(p.dirs, d)
|
||||
}
|
||||
// A folder's name is a prefix of everything in it, and a prefix sorts first.
|
||||
sort.Strings(p.dirs)
|
||||
sort.Slice(p.files, func(i, j int) bool { return p.files[i].name < p.files[j].name })
|
||||
return p, Result{}
|
||||
}
|
||||
|
||||
// entryName is an entry's name as its maker meant it. A zip made on Chinese
|
||||
// Windows stores names in the system code page (GBK) without the UTF-8 flag; a
|
||||
// name that is not valid UTF-8 is decoded as GB18030, GBK's superset. The flag
|
||||
// alone is no guide: archive/zip reports NonUTF8 for every name made without it,
|
||||
// which includes plain ASCII and macOS's UTF-8. Decoding comes before the
|
||||
// backslash below because a GBK trail byte can be 0x5C.
|
||||
func entryName(f *zip.File) string {
|
||||
name := f.Name
|
||||
if !utf8.ValidString(name) {
|
||||
if s, err := simplifiedchinese.GB18030.NewDecoder().String(name); err == nil {
|
||||
name = s
|
||||
}
|
||||
}
|
||||
return strings.ReplaceAll(name, `\`, "/")
|
||||
}
|
||||
|
||||
// cleanEntry cleans an entry name and reports false for one that must not be
|
||||
// extracted: a NUL, an absolute path, a drive letter, or a climb out of the
|
||||
// destination. It judges the name as TEXT, which is sound here because nothing
|
||||
// is resolved through it until checkTargets and extraction, and those go through
|
||||
// os.Root, which would refuse an escape anyway.
|
||||
func cleanEntry(raw string) (string, bool) {
|
||||
if raw == "" || strings.ContainsRune(raw, 0) || strings.HasPrefix(raw, "/") {
|
||||
return "", false
|
||||
}
|
||||
if len(raw) >= 2 && raw[1] == ':' && (raw[0]|0x20) >= 'a' && (raw[0]|0x20) <= 'z' {
|
||||
return "", false
|
||||
}
|
||||
name := path.Clean(raw)
|
||||
if name == ".." || strings.HasPrefix(name, "../") {
|
||||
return "", false
|
||||
}
|
||||
return name, true
|
||||
}
|
||||
|
||||
// checkTargets looks at what the server already has where the archive lands.
|
||||
// present is the archive's folders that exist on the server (they merge);
|
||||
// replaced is its files that do (conflicts). A folder the server lacks cannot
|
||||
// hold anything, so its contents are not looked up.
|
||||
func checkTargets(r *os.Root, dest string, p unzipPlan) (present, replaced map[string]bool, res Result) {
|
||||
present, replaced = map[string]bool{}, map[string]bool{}
|
||||
absent := func(name string) bool {
|
||||
parent := path.Dir(name)
|
||||
return parent != "." && !present[parent]
|
||||
}
|
||||
for _, d := range p.dirs {
|
||||
if absent(d) {
|
||||
continue
|
||||
}
|
||||
at := path.Join(dest, d)
|
||||
info, err := r.Lstat(at)
|
||||
switch {
|
||||
case errors.Is(err, fs.ErrNotExist):
|
||||
case err != nil:
|
||||
return nil, nil, failure(err, at)
|
||||
case info.Mode()&fs.ModeSymlink != 0:
|
||||
return nil, nil, typeConflict(at, "is a link on the server, and the archive has a folder there")
|
||||
case info.IsDir():
|
||||
present[d] = true
|
||||
default:
|
||||
return nil, nil, typeConflict(at, "is a file on the server, and the archive has a folder there")
|
||||
}
|
||||
}
|
||||
for _, zf := range p.files {
|
||||
if absent(zf.name) {
|
||||
continue
|
||||
}
|
||||
at := path.Join(dest, zf.name)
|
||||
info, err := r.Lstat(at)
|
||||
switch {
|
||||
case errors.Is(err, fs.ErrNotExist):
|
||||
case err != nil:
|
||||
return nil, nil, failure(err, at)
|
||||
case info.Mode().IsRegular():
|
||||
replaced[zf.name] = true
|
||||
case info.IsDir():
|
||||
return nil, nil, typeConflict(at, "is a folder on the server, and the archive has a file there")
|
||||
default:
|
||||
return nil, nil, typeConflict(at, "is a link or special file on the server, and the archive has a file there")
|
||||
}
|
||||
}
|
||||
return present, replaced, Result{}
|
||||
}
|
||||
|
||||
func typeConflict(at, why string) Result {
|
||||
return Result{Code: CodeTypeConflict, Entry: at, Error: fmt.Sprintf("%s %s; nothing was changed", at, why)}
|
||||
}
|
||||
|
||||
// extractAll writes every folder and file of p under staged, handed to the game
|
||||
// uid, and syncs each file.
|
||||
func extractAll(r *os.Root, staged string, p unzipPlan, progress func(done, total int64)) Result {
|
||||
for _, d := range p.dirs {
|
||||
at := path.Join(staged, d)
|
||||
if err := r.Mkdir(at, 0o755); err != nil {
|
||||
return unzipWriteFailure(err, d)
|
||||
}
|
||||
_ = ownWritten(r, at)
|
||||
}
|
||||
var done int64
|
||||
count := func(n int) {
|
||||
done += int64(n)
|
||||
if progress != nil {
|
||||
progress(done, p.bytes)
|
||||
}
|
||||
}
|
||||
for _, zf := range p.files {
|
||||
if res := extractOne(r, path.Join(staged, zf.name), zf, count); res.Code != "" {
|
||||
return res
|
||||
}
|
||||
}
|
||||
for _, d := range p.dirs {
|
||||
syncDir(r, path.Join(staged, d))
|
||||
}
|
||||
syncDir(r, staged)
|
||||
return Result{}
|
||||
}
|
||||
|
||||
func extractOne(r *os.Root, at string, zf zipFile, count func(int)) Result {
|
||||
invalid := func(err error) Result {
|
||||
return Result{Code: CodeArchiveInvalid, Entry: zf.name, Error: fmt.Sprintf(
|
||||
"%s in the archive is damaged: %v; nothing was changed", zf.name, err)}
|
||||
}
|
||||
src, err := zf.f.Open()
|
||||
if err != nil {
|
||||
return invalid(err)
|
||||
}
|
||||
defer src.Close()
|
||||
w, err := r.OpenFile(at, os.O_WRONLY|os.O_CREATE|os.O_EXCL, zf.mode)
|
||||
if err != nil {
|
||||
return unzipWriteFailure(err, zf.name)
|
||||
}
|
||||
if err := w.Chmod(zf.mode); err != nil {
|
||||
w.Close()
|
||||
return unzipWriteFailure(err, zf.name)
|
||||
}
|
||||
if _, err := io.Copy(countingWriter{w, count}, archiveReader{src}); err != nil {
|
||||
w.Close()
|
||||
var ae *archiveError
|
||||
if errors.As(err, &ae) {
|
||||
return invalid(ae.err)
|
||||
}
|
||||
return unzipWriteFailure(err, zf.name)
|
||||
}
|
||||
if err := syncWritten(w); err != nil {
|
||||
w.Close()
|
||||
return unzipWriteFailure(err, zf.name)
|
||||
}
|
||||
if err := w.Close(); err != nil {
|
||||
return unzipWriteFailure(err, zf.name)
|
||||
}
|
||||
_ = ownWritten(r, at)
|
||||
return Result{}
|
||||
}
|
||||
|
||||
// placeAll renames the extracted tree into dest. A folder the server lacks moves
|
||||
// whole; one it has is descended into. A file it has is first moved aside into
|
||||
// old, so undoing the journal puts it back.
|
||||
func placeAll(r *os.Root, dest, staged, old string, p unzipPlan, present, replaced map[string]bool) Result {
|
||||
kids := map[string][]string{}
|
||||
for _, d := range p.dirs {
|
||||
kids[path.Dir(d)] = append(kids[path.Dir(d)], d)
|
||||
}
|
||||
for _, zf := range p.files {
|
||||
kids[path.Dir(zf.name)] = append(kids[path.Dir(zf.name)], zf.name)
|
||||
}
|
||||
|
||||
type move struct{ from, to string }
|
||||
var journal []move
|
||||
mv := func(from, to string) error {
|
||||
if err := renameEntry(r, from, to); err != nil {
|
||||
return err
|
||||
}
|
||||
journal = append(journal, move{from, to})
|
||||
return nil
|
||||
}
|
||||
var place func(dir string) error
|
||||
place = func(dir string) error {
|
||||
for _, c := range kids[dir] {
|
||||
src, dst := path.Join(staged, c), path.Join(dest, c)
|
||||
switch {
|
||||
case p.isDir[c] && present[c]:
|
||||
if err := place(c); err != nil {
|
||||
return err
|
||||
}
|
||||
case replaced[c]:
|
||||
aside := path.Join(old, c)
|
||||
if err := r.MkdirAll(path.Dir(aside), 0o700); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := mv(dst, aside); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := mv(src, dst); err != nil {
|
||||
return err
|
||||
}
|
||||
default:
|
||||
if err := mv(src, dst); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if err := place("."); err != nil {
|
||||
for i := len(journal) - 1; i >= 0; i-- {
|
||||
_ = r.Rename(journal[i].to, journal[i].from)
|
||||
}
|
||||
return unzipWriteFailure(err, dest)
|
||||
}
|
||||
syncDir(r, dest)
|
||||
for d := range present {
|
||||
syncDir(r, path.Join(dest, d))
|
||||
}
|
||||
return Result{}
|
||||
}
|
||||
|
||||
// renameEntry is the rename placeAll moves with. A var so a test can fail one
|
||||
// part-way through and watch the journal undo the rest.
|
||||
var renameEntry = func(r *os.Root, from, to string) error { return r.Rename(from, to) }
|
||||
|
||||
// sweepUnzipTemps removes working folders a killed unzip left in dir.
|
||||
func sweepUnzipTemps(r *os.Root, dir string) {
|
||||
d, err := r.Open(dir)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
names, _ := d.Readdirnames(-1)
|
||||
d.Close()
|
||||
for _, n := range names {
|
||||
if isUnzipTemp(n) {
|
||||
_ = r.RemoveAll(path.Join(dir, n))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func isUnzipTemp(name string) bool {
|
||||
suffix, ok := strings.CutPrefix(name, unzipTempPrefix)
|
||||
if !ok || len(suffix) != 12 {
|
||||
return false
|
||||
}
|
||||
_, err := hex.DecodeString(suffix)
|
||||
return err == nil
|
||||
}
|
||||
|
||||
// unzipWriteFailure is writeFailure worded for an unzip, which by then has
|
||||
// changed nothing whatever the step that failed.
|
||||
func unzipWriteFailure(err error, name string) Result {
|
||||
res := writeFailure(err, name)
|
||||
if res.Code == CodeNoSpace {
|
||||
res.Error = "the server's volume filled up while extracting; nothing was changed"
|
||||
}
|
||||
return res
|
||||
}
|
||||
|
||||
// archiveError marks a failure reading an entry's bytes out of the archive, so
|
||||
// extractOne can tell a damaged archive from the volume failing underneath.
|
||||
type archiveError struct{ err error }
|
||||
|
||||
func (e *archiveError) Error() string { return "read archive: " + e.err.Error() }
|
||||
func (e *archiveError) Unwrap() error { return e.err }
|
||||
|
||||
type archiveReader struct{ r io.Reader }
|
||||
|
||||
func (a archiveReader) Read(p []byte) (int, error) {
|
||||
n, err := a.r.Read(p)
|
||||
if err != nil && err != io.EOF {
|
||||
err = &archiveError{err}
|
||||
}
|
||||
return n, err
|
||||
}
|
||||
|
||||
// countingWriter reports each write's length to add.
|
||||
type countingWriter struct {
|
||||
w io.Writer
|
||||
add func(int)
|
||||
}
|
||||
|
||||
func (c countingWriter) Write(p []byte) (int, error) {
|
||||
n, err := c.w.Write(p)
|
||||
c.add(n)
|
||||
return n, err
|
||||
}
|
||||
@@ -0,0 +1,539 @@
|
||||
package fileedit
|
||||
|
||||
import (
|
||||
"archive/zip"
|
||||
"bytes"
|
||||
"errors"
|
||||
"fmt"
|
||||
"hash/crc32"
|
||||
"io"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"syscall"
|
||||
"testing"
|
||||
|
||||
"golang.org/x/text/encoding/simplifiedchinese"
|
||||
)
|
||||
|
||||
// zent is one entry of a test archive: a deflated file, or a folder when the name
|
||||
// ends in "/".
|
||||
type zent struct {
|
||||
name string
|
||||
body string
|
||||
mode fs.FileMode // set on the header when non-zero
|
||||
flat bool // written without the UTF-8 flag, as old Windows tools do
|
||||
// raw writes body stored as-is under a header declaring size and crc, so a
|
||||
// test can build an archive whose entries lie about themselves.
|
||||
raw bool
|
||||
size uint64
|
||||
crc uint32
|
||||
}
|
||||
|
||||
func file(name, body string) zent { return zent{name: name, body: body} }
|
||||
|
||||
// lie is an entry declaring size bytes while holding body.
|
||||
func lie(name, body string, size uint64) zent {
|
||||
return zent{name: name, body: body, raw: true, size: size, crc: crc32.ChecksumIEEE([]byte(body))}
|
||||
}
|
||||
|
||||
func writeZip(t *testing.T, at string, entries ...zent) {
|
||||
t.Helper()
|
||||
var buf bytes.Buffer
|
||||
w := zip.NewWriter(&buf)
|
||||
for _, e := range entries {
|
||||
fh := &zip.FileHeader{Name: e.name, Method: zip.Deflate, NonUTF8: e.flat}
|
||||
if e.mode != 0 {
|
||||
fh.SetMode(e.mode)
|
||||
}
|
||||
var dst io.Writer
|
||||
var err error
|
||||
if e.raw {
|
||||
fh.Method = zip.Store
|
||||
fh.CRC32, fh.CompressedSize64, fh.UncompressedSize64 = e.crc, uint64(len(e.body)), e.size
|
||||
dst, err = w.CreateRaw(fh)
|
||||
} else {
|
||||
dst, err = w.CreateHeader(fh)
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("zip entry %q: %v", e.name, err)
|
||||
}
|
||||
if _, err := io.WriteString(dst, e.body); err != nil {
|
||||
t.Fatalf("zip entry %q: %v", e.name, err)
|
||||
}
|
||||
}
|
||||
if err := w.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(at, buf.Bytes(), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// tree is everything under dir: each file's content, "<dir>" for a folder and
|
||||
// "-> target" for a link. Comparing two trees is how a test says "nothing
|
||||
// changed", working folders included.
|
||||
func tree(t *testing.T, dir string) map[string]string {
|
||||
t.Helper()
|
||||
out := map[string]string{}
|
||||
err := filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
rel, _ := filepath.Rel(dir, p)
|
||||
switch {
|
||||
case rel == ".":
|
||||
case d.Type()&fs.ModeSymlink != 0:
|
||||
target, err := os.Readlink(p)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
out[rel] = "-> " + target
|
||||
case d.IsDir():
|
||||
out[rel] = "<dir>"
|
||||
default:
|
||||
b, err := os.ReadFile(p)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
out[rel] = string(b)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func assertSameTree(t *testing.T, before, after map[string]string) {
|
||||
t.Helper()
|
||||
if !reflect.DeepEqual(before, after) {
|
||||
t.Fatalf("the tree changed:\nbefore %v\nafter %v", before, after)
|
||||
}
|
||||
}
|
||||
|
||||
func unzipAt(t *testing.T, root, name string, overwrite bool) Result {
|
||||
t.Helper()
|
||||
return exec(t, root, Request{Op: OpUnzip, Path: name, Overwrite: overwrite})
|
||||
}
|
||||
|
||||
func TestUnzip(t *testing.T) {
|
||||
t.Run("extracts files and folders into the folder holding the archive", func(t *testing.T) {
|
||||
root, _ := worldRoot(t)
|
||||
if err := os.Mkdir(filepath.Join(root, "plugins"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
writeZip(t, filepath.Join(root, "plugins", "pack.zip"),
|
||||
zent{name: "Essentials/"},
|
||||
file("Essentials/config.yml", "locale: zh\n"),
|
||||
zent{name: "empty/"},
|
||||
file("./readme.txt", "hi"),
|
||||
file(`win\sub\a.txt`, "from windows"),
|
||||
)
|
||||
var owned []string
|
||||
prev := ownWritten
|
||||
ownWritten = func(_ *os.Root, name string) error { owned = append(owned, name); return nil }
|
||||
defer func() { ownWritten = prev }()
|
||||
|
||||
res := unzipAt(t, root, "plugins/pack.zip", false)
|
||||
if res.Code != "" || res.Files != 3 || res.Bytes != int64(len("locale: zh\n")+len("hi")+len("from windows")) {
|
||||
t.Fatalf("result = %+v", res)
|
||||
}
|
||||
got := tree(t, filepath.Join(root, "plugins"))
|
||||
delete(got, "pack.zip")
|
||||
want := map[string]string{
|
||||
"Essentials": "<dir>", "Essentials/config.yml": "locale: zh\n", "empty": "<dir>",
|
||||
"readme.txt": "hi", "win": "<dir>", "win/sub": "<dir>", "win/sub/a.txt": "from windows",
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("plugins/ = %v\nwant %v", got, want)
|
||||
}
|
||||
if info, _ := os.Stat(filepath.Join(root, "plugins", "readme.txt")); info.Mode().Perm() != 0o644 {
|
||||
t.Fatalf("mode = %v, want 0644", info.Mode().Perm())
|
||||
}
|
||||
// Every folder and file is handed to the game uid, while still in the
|
||||
// working folder: 4 folders and 3 files.
|
||||
if len(owned) != 7 {
|
||||
t.Fatalf("handed to the game uid: %v, want 7", owned)
|
||||
}
|
||||
for _, o := range owned {
|
||||
if !strings.HasPrefix(o, "plugins/"+unzipTempPrefix) {
|
||||
t.Fatalf("%s was chowned outside the working folder", o)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("an archive named in capitals extracts", func(t *testing.T) {
|
||||
root, _ := worldRoot(t)
|
||||
writeZip(t, filepath.Join(root, "PACK.ZIP"), file("a.txt", "a"))
|
||||
if res := unzipAt(t, root, "PACK.ZIP", false); res.Code != "" || res.Files != 1 {
|
||||
t.Fatalf("result = %+v", res)
|
||||
}
|
||||
if got := mustRead(t, filepath.Join(root, "a.txt")); got != "a" {
|
||||
t.Fatalf("a.txt = %q", got)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("__MACOSX is left out", func(t *testing.T) {
|
||||
root, _ := worldRoot(t)
|
||||
writeZip(t, filepath.Join(root, "mac.zip"), file("a.txt", "a"), zent{name: "__MACOSX/"}, file("__MACOSX/._a.txt", "fork"))
|
||||
if res := unzipAt(t, root, "mac.zip", false); res.Code != "" || res.Files != 1 {
|
||||
t.Fatalf("result = %+v", res)
|
||||
}
|
||||
assertAbsent(t, filepath.Join(root, "__MACOSX"))
|
||||
})
|
||||
|
||||
t.Run("a name made on Chinese Windows is decoded from GBK; UTF-8 without the flag is kept", func(t *testing.T) {
|
||||
root, _ := worldRoot(t)
|
||||
gbk, err := simplifiedchinese.GBK.NewEncoder().String("存档/说明.txt")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
writeZip(t, filepath.Join(root, "cn.zip"),
|
||||
zent{name: gbk, body: "中文", flat: true},
|
||||
zent{name: "macOS名字.txt", body: "utf8", flat: true},
|
||||
)
|
||||
if res := unzipAt(t, root, "cn.zip", false); res.Code != "" || res.Files != 2 {
|
||||
t.Fatalf("result = %+v", res)
|
||||
}
|
||||
if got := mustRead(t, filepath.Join(root, "存档", "说明.txt")); got != "中文" {
|
||||
t.Fatalf("存档/说明.txt = %q", got)
|
||||
}
|
||||
if got := mustRead(t, filepath.Join(root, "macOS名字.txt")); got != "utf8" {
|
||||
t.Fatalf("macOS名字.txt = %q", got)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("an executable entry stays executable; every other file is 0644", func(t *testing.T) {
|
||||
root, _ := worldRoot(t)
|
||||
writeZip(t, filepath.Join(root, "sh.zip"),
|
||||
zent{name: "start.sh", body: "#!/bin/sh\n", mode: 0o755},
|
||||
zent{name: "secret.txt", body: "x", mode: 0o600},
|
||||
)
|
||||
if res := unzipAt(t, root, "sh.zip", false); res.Code != "" {
|
||||
t.Fatalf("result = %+v", res)
|
||||
}
|
||||
for name, want := range map[string]fs.FileMode{"start.sh": 0o755, "secret.txt": 0o644} {
|
||||
if info, _ := os.Stat(filepath.Join(root, name)); info.Mode().Perm() != want {
|
||||
t.Errorf("%s mode = %v, want %v", name, info.Mode().Perm(), want)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("a name leading outside is archive_unsafe and nothing is written", func(t *testing.T) {
|
||||
for raw, entry := range map[string]string{
|
||||
"../evil.txt": "../evil.txt",
|
||||
"a/../../evil.txt": "a/../../evil.txt",
|
||||
`..\evil.txt`: "../evil.txt",
|
||||
"/evil.txt": "/evil.txt",
|
||||
"C:/evil.txt": "C:/evil.txt",
|
||||
`c:\evil.txt`: "c:/evil.txt",
|
||||
"nul\x00.txt": "nul\x00.txt",
|
||||
"a/..": "a/..",
|
||||
} {
|
||||
t.Run(entry, func(t *testing.T) {
|
||||
root, outside := worldRoot(t)
|
||||
writeZip(t, filepath.Join(root, "bad.zip"), file("ok.txt", "ok"), file(raw, "evil"))
|
||||
before := tree(t, root)
|
||||
res := unzipAt(t, root, "bad.zip", true)
|
||||
if res.Code != CodeArchiveUnsafe || res.Entry != entry {
|
||||
t.Fatalf("result = %+v; want archive_unsafe naming %q", res, entry)
|
||||
}
|
||||
assertSameTree(t, before, tree(t, root))
|
||||
assertAbsent(t, filepath.Join(outside, "evil.txt"))
|
||||
})
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("a link entry is archive_symlink and a pipe is archive_unsafe", func(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
mode fs.FileMode
|
||||
code string
|
||||
}{
|
||||
{fs.ModeSymlink | 0o777, CodeArchiveSymlink},
|
||||
{fs.ModeNamedPipe | 0o644, CodeArchiveUnsafe},
|
||||
} {
|
||||
root, _ := worldRoot(t)
|
||||
writeZip(t, filepath.Join(root, "odd.zip"),
|
||||
file("ok.txt", "ok"), zent{name: "odd", body: "../../outside/secret.txt", mode: tc.mode})
|
||||
before := tree(t, root)
|
||||
res := unzipAt(t, root, "odd.zip", true)
|
||||
if res.Code != tc.code || res.Entry != "odd" {
|
||||
t.Fatalf("%v: result = %+v; want %s naming odd", tc.mode, res, tc.code)
|
||||
}
|
||||
assertSameTree(t, before, tree(t, root))
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("a damaged or senseless archive is archive_invalid and nothing is written", func(t *testing.T) {
|
||||
for name, tc := range map[string]struct {
|
||||
entries []zent
|
||||
entry string
|
||||
}{
|
||||
"bytes past the declared size": {[]zent{file("ok.txt", "ok"), lie("lie.txt", "0123456789", 3)}, "lie.txt"},
|
||||
"bytes short of the declared size": {[]zent{file("ok.txt", "ok"), lie("lie.txt", "0123456789", 20)}, "lie.txt"},
|
||||
"a wrong checksum": {[]zent{file("ok.txt", "ok"),
|
||||
{name: "crc.txt", body: "0123456789", raw: true, size: 10, crc: crc32.ChecksumIEEE([]byte("0123456789")) + 1}}, "crc.txt"},
|
||||
"the same file twice": {[]zent{file("a.txt", "1"), file("a.txt", "2")}, "a.txt"},
|
||||
"a file and a folder at once": {[]zent{file("a", "1"), file("a/b.txt", "2")}, "a"},
|
||||
"nothing in it": {nil, ""},
|
||||
"nothing but __MACOSX": {[]zent{file("__MACOSX/._a", "fork")}, ""},
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
root, _ := worldRoot(t)
|
||||
writeZip(t, filepath.Join(root, "bad.zip"), tc.entries...)
|
||||
before := tree(t, root)
|
||||
res := unzipAt(t, root, "bad.zip", true)
|
||||
if res.Code != CodeArchiveInvalid || res.Entry != tc.entry {
|
||||
t.Fatalf("result = %+v; want archive_invalid naming %q", res, tc.entry)
|
||||
}
|
||||
assertSameTree(t, before, tree(t, root))
|
||||
})
|
||||
}
|
||||
t.Run("not a zip at all", func(t *testing.T) {
|
||||
root, _ := worldRoot(t)
|
||||
if err := os.WriteFile(filepath.Join(root, "fake.zip"), []byte("hello"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if res := unzipAt(t, root, "fake.zip", false); res.Code != CodeArchiveInvalid {
|
||||
t.Fatalf("result = %+v; want archive_invalid", res)
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
t.Run("the archive's own path is checked", func(t *testing.T) {
|
||||
root, outside := worldRoot(t)
|
||||
if err := os.Mkdir(filepath.Join(root, "dir.zip"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
writeZip(t, filepath.Join(outside, "x.zip"), file("evil.txt", "evil"))
|
||||
symlink(t, outside, filepath.Join(root, "escape-link"))
|
||||
for name, code := range map[string]string{
|
||||
"server.properties": CodeBadPath,
|
||||
"missing.zip": CodeNotFound,
|
||||
"dir.zip": CodeBadPath,
|
||||
"../outside/x.zip": CodeBadPath,
|
||||
"escape-link/x.zip": CodeBadPath,
|
||||
"config/../../x.zip": CodeBadPath,
|
||||
} {
|
||||
if res := unzipAt(t, root, name, true); res.Code != code {
|
||||
t.Errorf("%s: result = %+v; want %s", name, res, code)
|
||||
}
|
||||
}
|
||||
assertAbsent(t, filepath.Join(root, "evil.txt"))
|
||||
assertAbsent(t, filepath.Join(outside, "evil.txt"))
|
||||
})
|
||||
|
||||
t.Run("files already there are listed and nothing changes without overwrite", func(t *testing.T) {
|
||||
root, _ := worldRoot(t)
|
||||
writeZip(t, filepath.Join(root, "pack.zip"),
|
||||
file("server.properties", "motd=new\n"), file("config/paper.yml", "verbose: true\n"),
|
||||
file("config/new.yml", "new"), file("fresh/x.txt", "x"))
|
||||
before := tree(t, root)
|
||||
res := unzipAt(t, root, "pack.zip", false)
|
||||
if res.Code != CodeExists || res.ConflictCount != 2 ||
|
||||
!reflect.DeepEqual(res.Conflicts, []string{"config/paper.yml", "server.properties"}) {
|
||||
t.Fatalf("result = %+v; want exists listing config/paper.yml and server.properties", res)
|
||||
}
|
||||
assertSameTree(t, before, tree(t, root))
|
||||
})
|
||||
|
||||
t.Run("conflicts are named from the server's root when the archive sits in a folder", func(t *testing.T) {
|
||||
root, _ := worldRoot(t)
|
||||
writeZip(t, filepath.Join(root, "config", "pack.zip"), file("paper.yml", "verbose: true\n"))
|
||||
res := unzipAt(t, root, "config/pack.zip", false)
|
||||
if res.Code != CodeExists || !reflect.DeepEqual(res.Conflicts, []string{"config/paper.yml"}) {
|
||||
t.Fatalf("result = %+v", res)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("the list stops at MaxConflicts and the count does not", func(t *testing.T) {
|
||||
root, _ := worldRoot(t)
|
||||
var entries []zent
|
||||
for i := range MaxConflicts + 1 {
|
||||
name := fmt.Sprintf("f%03d.txt", i)
|
||||
if err := os.WriteFile(filepath.Join(root, name), []byte("old"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
entries = append(entries, file(name, "new"))
|
||||
}
|
||||
writeZip(t, filepath.Join(root, "many.zip"), entries...)
|
||||
res := unzipAt(t, root, "many.zip", false)
|
||||
if res.Code != CodeExists || res.ConflictCount != MaxConflicts+1 || len(res.Conflicts) != MaxConflicts ||
|
||||
res.Conflicts[0] != "f000.txt" || res.Conflicts[MaxConflicts-1] != "f199.txt" {
|
||||
t.Fatalf("code %q, count %d, %d listed (%v … %v)", res.Code, res.ConflictCount, len(res.Conflicts),
|
||||
res.Conflicts[:1], res.Conflicts[len(res.Conflicts)-1:])
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("overwrite replaces files, merges folders and keeps everything else", func(t *testing.T) {
|
||||
root, _ := worldRoot(t)
|
||||
if err := os.WriteFile(filepath.Join(root, "config", "keep.yml"), []byte("keep"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
writeZip(t, filepath.Join(root, "pack.zip"),
|
||||
file("server.properties", "motd=new\n"), file("config/paper.yml", "verbose: true\n"),
|
||||
file("config/new.yml", "new"), file("fresh/x.txt", "x"))
|
||||
res := unzipAt(t, root, "pack.zip", true)
|
||||
if res.Code != "" || res.Files != 4 {
|
||||
t.Fatalf("result = %+v", res)
|
||||
}
|
||||
got := tree(t, root)
|
||||
delete(got, "pack.zip")
|
||||
want := map[string]string{
|
||||
"server.properties": "motd=new\n", "config": "<dir>", "config/paper.yml": "verbose: true\n",
|
||||
"config/keep.yml": "keep", "config/new.yml": "new", "fresh": "<dir>", "fresh/x.txt": "x",
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("world = %v\nwant %v", got, want)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("a file where the server has a folder, the reverse, or a link is type_conflict even with overwrite", func(t *testing.T) {
|
||||
for name, tc := range map[string]struct {
|
||||
entry string
|
||||
want string
|
||||
}{
|
||||
"folder over a file": {"server.properties/x.txt", "server.properties"},
|
||||
"file over a folder": {"config", "config"},
|
||||
"folder over a link": {"escape-link/evil.txt", "escape-link"},
|
||||
"file over a link": {"planted.txt", "planted.txt"},
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
root, outside := worldRoot(t)
|
||||
symlink(t, outside, filepath.Join(root, "escape-link"))
|
||||
symlink(t, "server.properties", filepath.Join(root, "planted.txt"))
|
||||
writeZip(t, filepath.Join(root, "pack.zip"), file("ok.txt", "ok"), file(tc.entry, "evil"))
|
||||
before := tree(t, root)
|
||||
res := unzipAt(t, root, "pack.zip", true)
|
||||
if res.Code != CodeTypeConflict || res.Entry != tc.want {
|
||||
t.Fatalf("result = %+v; want type_conflict naming %s", res, tc.want)
|
||||
}
|
||||
assertSameTree(t, before, tree(t, root))
|
||||
assertAbsent(t, filepath.Join(outside, "evil.txt"))
|
||||
})
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("more than the volume has free is no_space before anything is written", func(t *testing.T) {
|
||||
root, _ := worldRoot(t)
|
||||
writeZip(t, filepath.Join(root, "pack.zip"), file("d/a.txt", "0123456789"), file("d/b.txt", "0123456789"))
|
||||
need := int64(20 + 3*unzipEntryOverhead) // two files and the folder d
|
||||
stubStatfs(t, uint64(need-1), 1<<30)
|
||||
before := tree(t, root)
|
||||
res := unzipAt(t, root, "pack.zip", false)
|
||||
if res.Code != CodeNoSpace || res.Need != need || res.Avail != need-1 {
|
||||
t.Fatalf("result = %+v; want no_space with need %d, avail %d", res, need, need-1)
|
||||
}
|
||||
assertSameTree(t, before, tree(t, root))
|
||||
stubStatfs(t, uint64(need), 1<<30)
|
||||
if res := unzipAt(t, root, "pack.zip", false); res.Code != "" {
|
||||
t.Fatalf("with exactly enough room: %+v", res)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("the volume filling up mid-way is no_space and changes nothing", func(t *testing.T) {
|
||||
root, _ := worldRoot(t)
|
||||
writeZip(t, filepath.Join(root, "pack.zip"), file("a.txt", "a"), file("server.properties", "motd=new\n"))
|
||||
calls := 0
|
||||
prev := syncWritten
|
||||
syncWritten = func(f *os.File) error {
|
||||
if calls++; calls == 2 {
|
||||
return syscall.ENOSPC
|
||||
}
|
||||
return f.Sync()
|
||||
}
|
||||
defer func() { syncWritten = prev }()
|
||||
before := tree(t, root)
|
||||
if res := unzipAt(t, root, "pack.zip", true); res.Code != CodeNoSpace {
|
||||
t.Fatalf("result = %+v; want no_space", res)
|
||||
}
|
||||
assertSameTree(t, before, tree(t, root))
|
||||
})
|
||||
|
||||
// The renames are the one step that touches the server's files; one failing
|
||||
// part-way must put back every file already moved, replaced ones included.
|
||||
t.Run("a rename failing part-way is undone, whichever it is", func(t *testing.T) {
|
||||
// config present: paper.yml aside + in, z.yml in; zeta in; server.properties aside + in.
|
||||
const moves = 6
|
||||
for failAt := 1; failAt <= moves; failAt++ {
|
||||
root, _ := worldRoot(t)
|
||||
writeZip(t, filepath.Join(root, "pack.zip"),
|
||||
file("config/paper.yml", "verbose: true\n"), file("config/z.yml", "z"),
|
||||
file("server.properties", "motd=new\n"), file("zeta/x.txt", "x"))
|
||||
calls := 0
|
||||
prev := renameEntry
|
||||
renameEntry = func(r *os.Root, from, to string) error {
|
||||
if calls++; calls == failAt {
|
||||
return errors.New("injected rename failure")
|
||||
}
|
||||
return r.Rename(from, to)
|
||||
}
|
||||
before := tree(t, root)
|
||||
res := unzipAt(t, root, "pack.zip", true)
|
||||
renameEntry = prev
|
||||
if res.Code == "" {
|
||||
t.Fatalf("rename %d failing: result = %+v; want a failure", failAt, res)
|
||||
}
|
||||
assertSameTree(t, before, tree(t, root))
|
||||
}
|
||||
// And with none failing, the count above is the real number of moves.
|
||||
root, _ := worldRoot(t)
|
||||
writeZip(t, filepath.Join(root, "pack.zip"),
|
||||
file("config/paper.yml", "verbose: true\n"), file("config/z.yml", "z"),
|
||||
file("server.properties", "motd=new\n"), file("zeta/x.txt", "x"))
|
||||
calls := 0
|
||||
prev := renameEntry
|
||||
renameEntry = func(r *os.Root, from, to string) error { calls++; return r.Rename(from, to) }
|
||||
defer func() { renameEntry = prev }()
|
||||
if res := unzipAt(t, root, "pack.zip", true); res.Code != "" || calls != moves {
|
||||
t.Fatalf("result = %+v, %d moves; want success in %d", res, calls, moves)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("a working folder a killed unzip left is cleared; a lookalike is kept", func(t *testing.T) {
|
||||
root, _ := worldRoot(t)
|
||||
stale := filepath.Join(root, unzipTempPrefix+"0123456789ab")
|
||||
if err := os.MkdirAll(filepath.Join(stale, "new"), 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Twelve characters that are not hex, and hex a byte short or long.
|
||||
lookalikes := []string{"notahexname!", "0123456789", "0123456789abcd"}
|
||||
for _, l := range lookalikes {
|
||||
if err := os.Mkdir(filepath.Join(root, unzipTempPrefix+l), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
writeZip(t, filepath.Join(root, "pack.zip"), file("a.txt", "a"))
|
||||
if res := unzipAt(t, root, "pack.zip", false); res.Code != "" {
|
||||
t.Fatalf("result = %+v", res)
|
||||
}
|
||||
assertAbsent(t, stale)
|
||||
for _, l := range lookalikes {
|
||||
if _, err := os.Stat(filepath.Join(root, unzipTempPrefix+l)); err != nil {
|
||||
t.Fatalf("lookalike %q removed: %v", l, err)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("progress climbs to the total", func(t *testing.T) {
|
||||
root, _ := worldRoot(t)
|
||||
big := strings.Repeat("x", 100<<10)
|
||||
writeZip(t, filepath.Join(root, "pack.zip"), file("a.bin", big), file("b.bin", big))
|
||||
var seen []int64
|
||||
var total int64
|
||||
res := exec(t, root, Request{Op: OpUnzip, Path: "pack.zip",
|
||||
Progress: func(done, all int64) { seen = append(seen, done); total = all }})
|
||||
if res.Code != "" || total != int64(2*len(big)) || len(seen) < 2 || seen[len(seen)-1] != total {
|
||||
t.Fatalf("result = %+v; progress %d calls ending at %v of %d", res, len(seen), seen[len(seen)-1:], total)
|
||||
}
|
||||
for i := 1; i < len(seen); i++ {
|
||||
if seen[i] < seen[i-1] {
|
||||
t.Fatalf("progress went backwards: %v", seen)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
Reference in new issue
Block a user