bluepython508/tsnet-proxy
1
0 Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Update

Browse Source
This commit is contained in:
bluepython508 committed 2026-10-22 12:27:19 +01:00
1 parent 3530afb854
commit 6903fcff27
2146 files changed
+112256 -190703
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

vendor/github.com/klauspost/compress/huff0/README.md
Generated Vendored
+89 -89
View File
@@ -1,89 +1,89 @@
# Huff0 entropy compression
This package provides Huff0 encoding and decoding as used in zstd.
[Huff0](https://github.com/Cyan4973/FiniteStateEntropy#new-generation-entropy-coders),
a Huffman codec designed for modern CPU, featuring OoO (Out of Order) operations on multiple ALU
(Arithmetic Logic Unit), achieving extremely fast compression and decompression speeds.
This can be used for compressing input with a lot of similar input values to the smallest number of bytes.
This does not perform any multi-byte [dictionary coding](https://en.wikipedia.org/wiki/Dictionary_coder) as LZ coders,
but it can be used as a secondary step to compressors (like Snappy) that does not do entropy encoding.
* [Godoc documentation](https://godoc.org/github.com/klauspost/compress/huff0)
## News
This is used as part of the [zstandard](https://github.com/klauspost/compress/tree/master/zstd#zstd) compression and decompression package.
This ensures that most functionality is well tested.
# Usage
This package provides a low level interface that allows to compress single independent blocks.
Each block is separate, and there is no built in integrity checks.
This means that the caller should keep track of block sizes and also do checksums if needed.
Compressing a block is done via the [`Compress1X`](https://godoc.org/github.com/klauspost/compress/huff0#Compress1X) and
[`Compress4X`](https://godoc.org/github.com/klauspost/compress/huff0#Compress4X) functions.
You must provide input and will receive the output and maybe an error.
These error values can be returned:
| Error | Description |
|---------------------|-----------------------------------------------------------------------------|
| `<nil>` | Everything ok, output is returned |
| `ErrIncompressible` | Returned when input is judged to be too hard to compress |
| `ErrUseRLE` | Returned from the compressor when the input is a single byte value repeated |
| `ErrTooBig` | Returned if the input block exceeds the maximum allowed size (128 Kib) |
| `(error)` | An internal error occurred. |
As can be seen above some of there are errors that will be returned even under normal operation so it is important to handle these.
To reduce allocations you can provide a [`Scratch`](https://godoc.org/github.com/klauspost/compress/huff0#Scratch) object
that can be re-used for successive calls. Both compression and decompression accepts a `Scratch` object, and the same
object can be used for both.
Be aware, that when re-using a `Scratch` object that the *output* buffer is also re-used, so if you are still using this
you must set the `Out` field in the scratch to nil. The same buffer is used for compression and decompression output.
The `Scratch` object will retain state that allows to re-use previous tables for encoding and decoding.
## Tables and re-use
Huff0 allows for reusing tables from the previous block to save space if that is expected to give better/faster results.
The Scratch object allows you to set a [`ReusePolicy`](https://godoc.org/github.com/klauspost/compress/huff0#ReusePolicy)
that controls this behaviour. See the documentation for details. This can be altered between each block.
Do however note that this information is *not* stored in the output block and it is up to the users of the package to
record whether [`ReadTable`](https://godoc.org/github.com/klauspost/compress/huff0#ReadTable) should be called,
based on the boolean reported back from the CompressXX call.
If you want to store the table separate from the data, you can access them as `OutData` and `OutTable` on the
[`Scratch`](https://godoc.org/github.com/klauspost/compress/huff0#Scratch) object.
## Decompressing
The first part of decoding is to initialize the decoding table through [`ReadTable`](https://godoc.org/github.com/klauspost/compress/huff0#ReadTable).
This will initialize the decoding tables.
You can supply the complete block to `ReadTable` and it will return the data part of the block
which can be given to the decompressor.
Decompressing is done by calling the [`Decompress1X`](https://godoc.org/github.com/klauspost/compress/huff0#Scratch.Decompress1X)
or [`Decompress4X`](https://godoc.org/github.com/klauspost/compress/huff0#Scratch.Decompress4X) function.
For concurrently decompressing content with a fixed table a stateless [`Decoder`](https://godoc.org/github.com/klauspost/compress/huff0#Decoder) can be requested which will remain correct as long as the scratch is unchanged. The capacity of the provided slice indicates the expected output size.
You must provide the output from the compression stage, at exactly the size you got back. If you receive an error back
your input was likely corrupted.
It is important to note that a successful decoding does *not* mean your output matches your original input.
There are no integrity checks, so relying on errors from the decompressor does not assure your data is valid.
# Contributing
Contributions are always welcome. Be aware that adding public functions will require good justification and breaking
changes will likely not be accepted. If in doubt open an issue before writing the PR.
# Huff0 entropy compression
This package provides Huff0 encoding and decoding as used in zstd.
[Huff0](https://github.com/Cyan4973/FiniteStateEntropy#new-generation-entropy-coders),
a Huffman codec designed for modern CPU, featuring OoO (Out of Order) operations on multiple ALU
(Arithmetic Logic Unit), achieving extremely fast compression and decompression speeds.
This can be used for compressing input with a lot of similar input values to the smallest number of bytes.
This does not perform any multi-byte [dictionary coding](https://en.wikipedia.org/wiki/Dictionary_coder) as LZ coders,
but it can be used as a secondary step to compressors (like Snappy) that does not do entropy encoding.
* [Godoc documentation](https://godoc.org/github.com/klauspost/compress/huff0)
## News
This is used as part of the [zstandard](https://github.com/klauspost/compress/tree/master/zstd#zstd) compression and decompression package.
This ensures that most functionality is well tested.
# Usage
This package provides a low level interface that allows to compress single independent blocks.
Each block is separate, and there is no built in integrity checks.
This means that the caller should keep track of block sizes and also do checksums if needed.
Compressing a block is done via the [`Compress1X`](https://godoc.org/github.com/klauspost/compress/huff0#Compress1X) and
[`Compress4X`](https://godoc.org/github.com/klauspost/compress/huff0#Compress4X) functions.
You must provide input and will receive the output and maybe an error.
These error values can be returned:
| Error | Description |
|---------------------|-----------------------------------------------------------------------------|
| `<nil>` | Everything ok, output is returned |
| `ErrIncompressible` | Returned when input is judged to be too hard to compress |
| `ErrUseRLE` | Returned from the compressor when the input is a single byte value repeated |
| `ErrTooBig` | Returned if the input block exceeds the maximum allowed size (128 Kib) |
| `(error)` | An internal error occurred. |
As can be seen above some of there are errors that will be returned even under normal operation so it is important to handle these.
To reduce allocations you can provide a [`Scratch`](https://godoc.org/github.com/klauspost/compress/huff0#Scratch) object
that can be re-used for successive calls. Both compression and decompression accepts a `Scratch` object, and the same
object can be used for both.
Be aware, that when re-using a `Scratch` object that the *output* buffer is also re-used, so if you are still using this
you must set the `Out` field in the scratch to nil. The same buffer is used for compression and decompression output.
The `Scratch` object will retain state that allows to re-use previous tables for encoding and decoding.
## Tables and re-use
Huff0 allows for reusing tables from the previous block to save space if that is expected to give better/faster results.
The Scratch object allows you to set a [`ReusePolicy`](https://godoc.org/github.com/klauspost/compress/huff0#ReusePolicy)
that controls this behaviour. See the documentation for details. This can be altered between each block.
Do however note that this information is *not* stored in the output block and it is up to the users of the package to
record whether [`ReadTable`](https://godoc.org/github.com/klauspost/compress/huff0#ReadTable) should be called,
based on the boolean reported back from the CompressXX call.
If you want to store the table separate from the data, you can access them as `OutData` and `OutTable` on the
[`Scratch`](https://godoc.org/github.com/klauspost/compress/huff0#Scratch) object.
## Decompressing
The first part of decoding is to initialize the decoding table through [`ReadTable`](https://godoc.org/github.com/klauspost/compress/huff0#ReadTable).
This will initialize the decoding tables.
You can supply the complete block to `ReadTable` and it will return the data part of the block
which can be given to the decompressor.
Decompressing is done by calling the [`Decompress1X`](https://godoc.org/github.com/klauspost/compress/huff0#Scratch.Decompress1X)
or [`Decompress4X`](https://godoc.org/github.com/klauspost/compress/huff0#Scratch.Decompress4X) function.
For concurrently decompressing content with a fixed table a stateless [`Decoder`](https://godoc.org/github.com/klauspost/compress/huff0#Decoder) can be requested which will remain correct as long as the scratch is unchanged. The capacity of the provided slice indicates the expected output size.
You must provide the output from the compression stage, at exactly the size you got back. If you receive an error back
your input was likely corrupted.
It is important to note that a successful decoding does *not* mean your output matches your original input.
There are no integrity checks, so relying on errors from the decompressor does not assure your data is valid.
# Contributing
Contributions are always welcome. Be aware that adding public functions will require good justification and breaking
changes will likely not be accepted. If in doubt open an issue before writing the PR.
vendor/github.com/klauspost/compress/huff0/build_table.go
Generated Vendored
+168
View File
@@ -0,0 +1,168 @@
package huff0
import "errors"
// BuildCTable builds a Huffman compression table from a precomputed symbol
// histogram and installs it as the previous (reuse) table on s.
//
// After this call:
// - EstimateSize/CanUseTable can probe the table against other histograms.
// - Compress1X/Compress4X with Reuse = ReusePolicyMust will encode without
// emitting a new table header.
// - TransferCTable can hand the table to a sibling Scratch.
//
// count[i] is the number of occurrences of symbol i. The histogram must have
// at least 2 distinct non-zero symbols; ErrUseRLE is returned for a single
// symbol and an error is returned for an empty histogram.
func (s *Scratch) BuildCTable(count *[256]uint32) error {
if s == nil {
return errors.New("huff0: BuildCTable on nil Scratch")
}
if count == nil {
return errors.New("huff0: nil count passed to BuildCTable")
}
var err error
s, err = s.prepare(nil)
if err != nil {
return err
}
s.count = *count
var total, maxCount int
var symLen uint16
for i, v := range s.count {
total += int(v)
if int(v) > maxCount {
maxCount = int(v)
}
if v != 0 {
symLen = uint16(i) + 1
}
}
if total == 0 {
return errors.New("huff0: empty histogram")
}
if symLen < 2 || maxCount == total {
return ErrUseRLE
}
// huff0's internal rank table assumes total ≤ BlockSizeMax (it uses
// highBit32(count+1) + 1 as a rank index into a fixed-size array).
// Histograms summed across multiple blocks can exceed that; scale the
// counts down preserving the distribution. Non-zero entries round up so
// rare symbols stay representable.
if total > BlockSizeMax {
shift := uint(0)
for total>>shift > BlockSizeMax {
shift++
}
round := uint32(1<<shift) - 1
var newTotal, newMax int
for i, v := range s.count {
if v == 0 {
continue
}
scaled := (v + round) >> shift
if scaled == 0 {
scaled = 1
}
s.count[i] = scaled
newTotal += int(scaled)
if int(scaled) > newMax {
newMax = int(scaled)
}
}
total = newTotal
maxCount = newMax
if maxCount == total {
return ErrUseRLE
}
}
s.symbolLen = symLen
s.maxCount = maxCount
s.srcLen = total
if err := s.buildCTable(); err != nil {
return err
}
if cap(s.prevTable) < len(s.cTable) {
s.prevTable = make(cTable, 0, maxSymbolValue+1)
}
s.prevTable = s.prevTable[:len(s.cTable)]
copy(s.prevTable, s.cTable)
s.prevTableLog = s.actualTableLog
// Force the next Compress* to recount from real input.
s.clearCount = true
s.maxCount = 0
return nil
}
// EstimateSize returns an estimated compressed payload size in bytes for the
// supplied histogram using the table currently stored in prevTable. It returns
// -1 when the table cannot encode every non-zero symbol of hist (i.e. when
// CanUseTable would return false). The estimate excludes the table header.
func (s *Scratch) EstimateSize(hist *[256]uint32) int {
if s == nil || hist == nil || len(s.prevTable) == 0 {
return -1
}
pt := s.prevTable
nbBits := uint32(7)
for i, v := range hist {
if v == 0 {
continue
}
if i >= len(pt) || pt[i].nBits == 0 {
return -1
}
nbBits += uint32(pt[i].nBits) * v
}
return int(nbBits >> 3)
}
// CanUseTable reports whether the table in prevTable can encode every
// non-zero symbol present in hist.
func (s *Scratch) CanUseTable(hist *[256]uint32) bool {
if s == nil || hist == nil || len(s.prevTable) == 0 {
return false
}
pt := s.prevTable
for i, v := range hist {
if v == 0 {
continue
}
if i >= len(pt) || pt[i].nBits == 0 {
return false
}
}
return true
}
// AppendTable serializes the table currently stored in prevTable (e.g. as
// installed by BuildCTable or carried over from a previous Compress call)
// into a self-delimiting zstd-style header and appends it to dst. The
// returned slice can be parsed back by ReadTable.
func (s *Scratch) AppendTable(dst []byte) ([]byte, error) {
if s == nil || len(s.prevTable) == 0 {
return dst, errors.New("huff0: AppendTable with empty table")
}
// cTable.write reads s.actualTableLog, s.symbolLen, s.huffWeight, s.fse
// and writes into s.Out. Save/restore Out so we don't disturb in-flight
// compression buffers.
saveOut := s.Out
saveTL := s.actualTableLog
saveSL := s.symbolLen
if s.fse == nil {
// Lazily init in case AppendTable is called on a fresh Scratch.
if _, err := s.prepare(nil); err != nil {
return dst, err
}
saveOut = s.Out
}
s.Out = s.Out[:0]
s.actualTableLog = s.prevTableLog
s.symbolLen = uint16(len(s.prevTable))
if err := s.prevTable.write(s); err != nil {
s.Out, s.actualTableLog, s.symbolLen = saveOut, saveTL, saveSL
return dst, err
}
dst = append(dst, s.Out...)
s.Out, s.actualTableLog, s.symbolLen = saveOut, saveTL, saveSL
return dst, nil
}
vendor/github.com/klauspost/compress/huff0/decompress_amd64.go
Generated Vendored
+15 -189
View File
@@ -1,223 +1,49 @@
//go:build amd64 && !appengine && !noasm && gc
// +build amd64,!appengine,!noasm,gc
// This file contains the specialisation of Decoder.Decompress4X
// and Decoder.Decompress1X that use an asm implementation of thir main loops.
// amd64 stubs and dispatch for the asm loops used by decompress_asm.go.
package huff0
import (
"errors"
"fmt"
"github.com/klauspost/compress/internal/cpuinfo"
)
// decompress4x_main_loop_x86 is an x86 assembler implementation
// decompress4x_main_loop_amd64 is an x86 assembler implementation
// of Decompress4X when tablelog > 8.
//
//go:noescape
func decompress4x_main_loop_amd64(ctx *decompress4xContext)
// decompress4x_8b_loop_x86 is an x86 assembler implementation
// decompress4x_8b_main_loop_amd64 is an x86 assembler implementation
// of Decompress4X when tablelog <= 8 which decodes 4 entries
// per loop.
//
//go:noescape
func decompress4x_8b_main_loop_amd64(ctx *decompress4xContext)
// fallback8BitSize is the size where using Go version is faster.
const fallback8BitSize = 800
type decompress4xContext struct {
pbr *[4]bitReaderShifted
peekBits uint8
out *byte
dstEvery int
tbl *dEntrySingle
decoded int
limit *byte
}
// Decompress4X will decompress a 4X encoded stream.
// The length of the supplied input must match the end of a block exactly.
// The *capacity* of the dst slice must match the destination size of
// the uncompressed data exactly.
func (d *Decoder) Decompress4X(dst, src []byte) ([]byte, error) {
if len(d.dt.single) == 0 {
return nil, errors.New("no table loaded")
}
if len(src) < 6+(4*1) {
return nil, errors.New("input too small")
}
use8BitTables := d.actualTableLog <= 8
if cap(dst) < fallback8BitSize && use8BitTables {
return d.decompress4X8bit(dst, src)
}
var br [4]bitReaderShifted
// Decode "jump table"
start := 6
for i := range 3 {
length := int(src[i*2]) | (int(src[i*2+1]) << 8)
if start+length >= len(src) {
return nil, errors.New("truncated input (or invalid offset)")
}
err := br[i].init(src[start : start+length])
if err != nil {
return nil, err
}
start += length
}
err := br[3].init(src[start:])
if err != nil {
return nil, err
}
// destination, offset to match first output
dstSize := cap(dst)
dst = dst[:dstSize]
out := dst
dstEvery := (dstSize + 3) / 4
const tlSize = 1 << tableLogMax
const tlMask = tlSize - 1
single := d.dt.single[:tlSize]
var decoded int
if len(out) > 4*4 && !(br[0].off < 4 || br[1].off < 4 || br[2].off < 4 || br[3].off < 4) {
ctx := decompress4xContext{
pbr: &br,
peekBits: uint8((64 - d.actualTableLog) & 63), // see: bitReaderShifted.peekBitsFast()
out: &out[0],
dstEvery: dstEvery,
tbl: &single[0],
limit: &out[dstEvery-4], // Always stop decoding when first buffer gets here to avoid writing OOB on last.
}
if use8BitTables {
decompress4x_8b_main_loop_amd64(&ctx)
} else {
decompress4x_main_loop_amd64(&ctx)
}
decoded = ctx.decoded
out = out[decoded/4:]
}
// Decode remaining.
remainBytes := dstEvery - (decoded / 4)
for i := range br {
offset := dstEvery * i
endsAt := min(offset+remainBytes, len(out))
br := &br[i]
bitsLeft := br.remaining()
for bitsLeft > 0 {
br.fill()
if offset >= endsAt {
return nil, errors.New("corruption detected: stream overrun 4")
}
// Read value and increment offset.
val := br.peekBitsFast(d.actualTableLog)
v := single[val&tlMask].entry
nBits := uint8(v)
br.advance(nBits)
bitsLeft -= uint(nBits)
out[offset] = uint8(v >> 8)
offset++
}
if offset != endsAt {
return nil, fmt.Errorf("corruption detected: short output block %d, end %d != %d", i, offset, endsAt)
}
decoded += offset - dstEvery*i
err = br.close()
if err != nil {
return nil, err
}
}
if dstSize != decoded {
return nil, errors.New("corruption detected: short output block")
}
return dst, nil
}
// decompress4x_main_loop_x86 is an x86 assembler implementation
// decompress1x_main_loop_amd64 is an x86 assembler implementation
// of Decompress1X when tablelog > 8.
//
//go:noescape
func decompress1x_main_loop_amd64(ctx *decompress1xContext)
// decompress4x_main_loop_x86 is an x86 with BMI2 assembler implementation
// decompress1x_main_loop_bmi2 is an x86 with BMI2 assembler implementation
// of Decompress1X when tablelog > 8.
//
//go:noescape
func decompress1x_main_loop_bmi2(ctx *decompress1xContext)
type decompress1xContext struct {
pbr *bitReaderShifted
peekBits uint8
out *byte
outCap int
tbl *dEntrySingle
decoded int
func decompress4x_main_loop_asm(ctx *decompress4xContext) {
decompress4x_main_loop_amd64(ctx)
}
// Error reported by asm implementations
const error_max_decoded_size_exeeded = -1
func decompress4x_8b_main_loop_asm(ctx *decompress4xContext) {
decompress4x_8b_main_loop_amd64(ctx)
}
// Decompress1X will decompress a 1X encoded stream.
// The cap of the output buffer will be the maximum decompressed size.
// The length of the supplied input must match the end of a block exactly.
func (d *Decoder) Decompress1X(dst, src []byte) ([]byte, error) {
if len(d.dt.single) == 0 {
return nil, errors.New("no table loaded")
func decompress1x_main_loop_asm(ctx *decompress1xContext) {
if cpuinfo.HasBMI2() {
decompress1x_main_loop_bmi2(ctx)
} else {
decompress1x_main_loop_amd64(ctx)
}
var br bitReaderShifted
err := br.init(src)
if err != nil {
return dst, err
}
maxDecodedSize := cap(dst)
dst = dst[:maxDecodedSize]
const tlSize = 1 << tableLogMax
const tlMask = tlSize - 1
if maxDecodedSize >= 4 {
ctx := decompress1xContext{
pbr: &br,
out: &dst[0],
outCap: maxDecodedSize,
peekBits: uint8((64 - d.actualTableLog) & 63), // see: bitReaderShifted.peekBitsFast()
tbl: &d.dt.single[0],
}
if cpuinfo.HasBMI2() {
decompress1x_main_loop_bmi2(&ctx)
} else {
decompress1x_main_loop_amd64(&ctx)
}
if ctx.decoded == error_max_decoded_size_exeeded {
return nil, ErrMaxDecodedSizeExceeded
}
dst = dst[:ctx.decoded]
}
// br < 8, so uint8 is fine
bitsLeft := uint8(br.off)*8 + 64 - br.bitsRead
for bitsLeft > 0 {
br.fill()
if len(dst) >= maxDecodedSize {
br.close()
return nil, ErrMaxDecodedSizeExceeded
}
v := d.dt.single[br.peekBitsFast(d.actualTableLog)&tlMask]
nBits := uint8(v.entry)
br.advance(nBits)
bitsLeft -= nBits
dst = append(dst, uint8(v.entry>>8))
}
return dst, br.close()
}
vendor/github.com/klauspost/compress/huff0/decompress_amd64.s
Generated Vendored
+105 -105
View File
@@ -1,6 +1,6 @@
// Code generated by command: go run gen.go -out ../decompress_amd64.s -pkg=huff0. DO NOT EDIT.
// Code generated by command: go run gen.go -out ../decompress.s -arch amd64,arm64 -pkg=huff0. DO NOT EDIT.
//go:build amd64 && !appengine && !noasm && gc
//go:build !appengine && !noasm && gc
// func decompress4x_main_loop_amd64(ctx *decompress4xContext)
TEXT ·decompress4x_main_loop_amd64(SB), $0-8
@@ -47,12 +47,12 @@ skip_fill0:
SHRQ CL, R13
// v0 := table[val0&mask]
MOVW (R9)(R13*2), CX
MOVWQZX (R9)(R13*2), CX
// br0.advance(uint8(v0.entry)
MOVB CH, AL
SHLQ CL, R11
ADDB CL, R12
MOVBLZX CH, AX
SHLQ CL, R11
ADDB CL, R12
// val1 := br0.peekTopBits(peekBits)
MOVQ DI, CX
@@ -60,7 +60,7 @@ skip_fill0:
SHRQ CL, R13
// v1 := table[val1&mask]
MOVW (R9)(R13*2), CX
MOVWQZX (R9)(R13*2), CX
// br0.advance(uint8(v1.entry))
MOVB CH, AH
@@ -104,12 +104,12 @@ skip_fill1:
SHRQ CL, R13
// v0 := table[val0&mask]
MOVW (R9)(R13*2), CX
MOVWQZX (R9)(R13*2), CX
// br1.advance(uint8(v0.entry)
MOVB CH, AL
SHLQ CL, R11
ADDB CL, R12
MOVBLZX CH, AX
SHLQ CL, R11
ADDB CL, R12
// val1 := br1.peekTopBits(peekBits)
MOVQ DI, CX
@@ -117,7 +117,7 @@ skip_fill1:
SHRQ CL, R13
// v1 := table[val1&mask]
MOVW (R9)(R13*2), CX
MOVWQZX (R9)(R13*2), CX
// br1.advance(uint8(v1.entry))
MOVB CH, AH
@@ -161,12 +161,12 @@ skip_fill2:
SHRQ CL, R13
// v0 := table[val0&mask]
MOVW (R9)(R13*2), CX
MOVWQZX (R9)(R13*2), CX
// br2.advance(uint8(v0.entry)
MOVB CH, AL
SHLQ CL, R11
ADDB CL, R12
MOVBLZX CH, AX
SHLQ CL, R11
ADDB CL, R12
// val1 := br2.peekTopBits(peekBits)
MOVQ DI, CX
@@ -174,7 +174,7 @@ skip_fill2:
SHRQ CL, R13
// v1 := table[val1&mask]
MOVW (R9)(R13*2), CX
MOVWQZX (R9)(R13*2), CX
// br2.advance(uint8(v1.entry))
MOVB CH, AH
@@ -218,12 +218,12 @@ skip_fill3:
SHRQ CL, R13
// v0 := table[val0&mask]
MOVW (R9)(R13*2), CX
MOVWQZX (R9)(R13*2), CX
// br3.advance(uint8(v0.entry)
MOVB CH, AL
SHLQ CL, R11
ADDB CL, R12
MOVBLZX CH, AX
SHLQ CL, R11
ADDB CL, R12
// val1 := br3.peekTopBits(peekBits)
MOVQ DI, CX
@@ -231,7 +231,7 @@ skip_fill3:
SHRQ CL, R13
// v1 := table[val1&mask]
MOVW (R9)(R13*2), CX
MOVWQZX (R9)(R13*2), CX
// br3.advance(uint8(v1.entry))
MOVB CH, AH
@@ -259,13 +259,13 @@ skip_fill3:
// func decompress4x_8b_main_loop_amd64(ctx *decompress4xContext)
TEXT ·decompress4x_8b_main_loop_amd64(SB), $0-8
// Preload values
MOVQ ctx+0(FP), CX
MOVBQZX 8(CX), DI
MOVQ 16(CX), BX
MOVQ 48(CX), SI
MOVQ 24(CX), R8
MOVQ 32(CX), R9
MOVQ (CX), R10
MOVQ ctx+0(FP), AX
MOVBQZX 8(AX), DI
MOVQ 16(AX), BX
MOVQ 48(AX), SI
MOVQ 24(AX), R8
MOVQ 32(AX), R9
MOVQ (AX), R10
// Main loop
main_loop:
@@ -278,35 +278,35 @@ main_loop:
MOVBQZX 40(R10), R12
CMPQ R12, $0x20
JBE skip_fill0
MOVQ 24(R10), R13
MOVQ 24(R10), AX
SUBQ $0x20, R12
SUBQ $0x04, R13
MOVQ (R10), R14
SUBQ $0x04, AX
MOVQ (R10), R13
// b.value |= uint64(low) << (b.bitsRead & 63)
MOVL (R13)(R14*1), R14
MOVL (AX)(R13*1), R13
MOVQ R12, CX
SHLQ CL, R14
MOVQ R13, 24(R10)
ORQ R14, R11
SHLQ CL, R13
MOVQ AX, 24(R10)
ORQ R13, R11
// exhausted += (br0.off < 4)
CMPQ R13, $0x04
CMPQ AX, $0x04
ADCB $+0, DL
skip_fill0:
// val0 := br0.peekTopBits(peekBits)
MOVQ R11, R13
MOVQ R11, AX
MOVQ DI, CX
SHRQ CL, R13
SHRQ CL, AX
// v0 := table[val0&mask]
MOVW (R9)(R13*2), CX
MOVWQZX (R9)(AX*2), CX
// br0.advance(uint8(v0.entry)
MOVB CH, AL
SHLQ CL, R11
ADDB CL, R12
MOVBLZX CH, AX
SHLQ CL, R11
ADDB CL, R12
// val1 := br0.peekTopBits(peekBits)
MOVQ R11, R13
@@ -314,7 +314,7 @@ skip_fill0:
SHRQ CL, R13
// v1 := table[val0&mask]
MOVW (R9)(R13*2), CX
MOVWQZX (R9)(R13*2), CX
// br0.advance(uint8(v1.entry)
MOVB CH, AH
@@ -328,7 +328,7 @@ skip_fill0:
SHRQ CL, R13
// v2 := table[val0&mask]
MOVW (R9)(R13*2), CX
MOVWQZX (R9)(R13*2), CX
// br0.advance(uint8(v2.entry)
MOVB CH, AH
@@ -341,7 +341,7 @@ skip_fill0:
SHRQ CL, R13
// v3 := table[val0&mask]
MOVW (R9)(R13*2), CX
MOVWQZX (R9)(R13*2), CX
// br0.advance(uint8(v3.entry)
MOVB CH, AL
@@ -365,35 +365,35 @@ skip_fill0:
MOVBQZX 88(R10), R12
CMPQ R12, $0x20
JBE skip_fill1
MOVQ 72(R10), R13
MOVQ 72(R10), AX
SUBQ $0x20, R12
SUBQ $0x04, R13
MOVQ 48(R10), R14
SUBQ $0x04, AX
MOVQ 48(R10), R13
// b.value |= uint64(low) << (b.bitsRead & 63)
MOVL (R13)(R14*1), R14
MOVL (AX)(R13*1), R13
MOVQ R12, CX
SHLQ CL, R14
MOVQ R13, 72(R10)
ORQ R14, R11
SHLQ CL, R13
MOVQ AX, 72(R10)
ORQ R13, R11
// exhausted += (br1.off < 4)
CMPQ R13, $0x04
CMPQ AX, $0x04
ADCB $+0, DL
skip_fill1:
// val0 := br1.peekTopBits(peekBits)
MOVQ R11, R13
MOVQ R11, AX
MOVQ DI, CX
SHRQ CL, R13
SHRQ CL, AX
// v0 := table[val0&mask]
MOVW (R9)(R13*2), CX
MOVWQZX (R9)(AX*2), CX
// br1.advance(uint8(v0.entry)
MOVB CH, AL
SHLQ CL, R11
ADDB CL, R12
MOVBLZX CH, AX
SHLQ CL, R11
ADDB CL, R12
// val1 := br1.peekTopBits(peekBits)
MOVQ R11, R13
@@ -401,7 +401,7 @@ skip_fill1:
SHRQ CL, R13
// v1 := table[val0&mask]
MOVW (R9)(R13*2), CX
MOVWQZX (R9)(R13*2), CX
// br1.advance(uint8(v1.entry)
MOVB CH, AH
@@ -415,7 +415,7 @@ skip_fill1:
SHRQ CL, R13
// v2 := table[val0&mask]
MOVW (R9)(R13*2), CX
MOVWQZX (R9)(R13*2), CX
// br1.advance(uint8(v2.entry)
MOVB CH, AH
@@ -428,7 +428,7 @@ skip_fill1:
SHRQ CL, R13
// v3 := table[val0&mask]
MOVW (R9)(R13*2), CX
MOVWQZX (R9)(R13*2), CX
// br1.advance(uint8(v3.entry)
MOVB CH, AL
@@ -452,35 +452,35 @@ skip_fill1:
MOVBQZX 136(R10), R12
CMPQ R12, $0x20
JBE skip_fill2
MOVQ 120(R10), R13
MOVQ 120(R10), AX
SUBQ $0x20, R12
SUBQ $0x04, R13
MOVQ 96(R10), R14
SUBQ $0x04, AX
MOVQ 96(R10), R13
// b.value |= uint64(low) << (b.bitsRead & 63)
MOVL (R13)(R14*1), R14
MOVL (AX)(R13*1), R13
MOVQ R12, CX
SHLQ CL, R14
MOVQ R13, 120(R10)
ORQ R14, R11
SHLQ CL, R13
MOVQ AX, 120(R10)
ORQ R13, R11
// exhausted += (br2.off < 4)
CMPQ R13, $0x04
CMPQ AX, $0x04
ADCB $+0, DL
skip_fill2:
// val0 := br2.peekTopBits(peekBits)
MOVQ R11, R13
MOVQ R11, AX
MOVQ DI, CX
SHRQ CL, R13
SHRQ CL, AX
// v0 := table[val0&mask]
MOVW (R9)(R13*2), CX
MOVWQZX (R9)(AX*2), CX
// br2.advance(uint8(v0.entry)
MOVB CH, AL
SHLQ CL, R11
ADDB CL, R12
MOVBLZX CH, AX
SHLQ CL, R11
ADDB CL, R12
// val1 := br2.peekTopBits(peekBits)
MOVQ R11, R13
@@ -488,7 +488,7 @@ skip_fill2:
SHRQ CL, R13
// v1 := table[val0&mask]
MOVW (R9)(R13*2), CX
MOVWQZX (R9)(R13*2), CX
// br2.advance(uint8(v1.entry)
MOVB CH, AH
@@ -502,7 +502,7 @@ skip_fill2:
SHRQ CL, R13
// v2 := table[val0&mask]
MOVW (R9)(R13*2), CX
MOVWQZX (R9)(R13*2), CX
// br2.advance(uint8(v2.entry)
MOVB CH, AH
@@ -515,7 +515,7 @@ skip_fill2:
SHRQ CL, R13
// v3 := table[val0&mask]
MOVW (R9)(R13*2), CX
MOVWQZX (R9)(R13*2), CX
// br2.advance(uint8(v3.entry)
MOVB CH, AL
@@ -539,35 +539,35 @@ skip_fill2:
MOVBQZX 184(R10), R12
CMPQ R12, $0x20
JBE skip_fill3
MOVQ 168(R10), R13
MOVQ 168(R10), AX
SUBQ $0x20, R12
SUBQ $0x04, R13
MOVQ 144(R10), R14
SUBQ $0x04, AX
MOVQ 144(R10), R13
// b.value |= uint64(low) << (b.bitsRead & 63)
MOVL (R13)(R14*1), R14
MOVL (AX)(R13*1), R13
MOVQ R12, CX
SHLQ CL, R14
MOVQ R13, 168(R10)
ORQ R14, R11
SHLQ CL, R13
MOVQ AX, 168(R10)
ORQ R13, R11
// exhausted += (br3.off < 4)
CMPQ R13, $0x04
CMPQ AX, $0x04
ADCB $+0, DL
skip_fill3:
// val0 := br3.peekTopBits(peekBits)
MOVQ R11, R13
MOVQ R11, AX
MOVQ DI, CX
SHRQ CL, R13
SHRQ CL, AX
// v0 := table[val0&mask]
MOVW (R9)(R13*2), CX
MOVWQZX (R9)(AX*2), CX
// br3.advance(uint8(v0.entry)
MOVB CH, AL
SHLQ CL, R11
ADDB CL, R12
MOVBLZX CH, AX
SHLQ CL, R11
ADDB CL, R12
// val1 := br3.peekTopBits(peekBits)
MOVQ R11, R13
@@ -575,7 +575,7 @@ skip_fill3:
SHRQ CL, R13
// v1 := table[val0&mask]
MOVW (R9)(R13*2), CX
MOVWQZX (R9)(R13*2), CX
// br3.advance(uint8(v1.entry)
MOVB CH, AH
@@ -589,7 +589,7 @@ skip_fill3:
SHRQ CL, R13
// v2 := table[val0&mask]
MOVW (R9)(R13*2), CX
MOVWQZX (R9)(R13*2), CX
// br3.advance(uint8(v2.entry)
MOVB CH, AH
@@ -602,7 +602,7 @@ skip_fill3:
SHRQ CL, R13
// v3 := table[val0&mask]
MOVW (R9)(R13*2), CX
MOVWQZX (R9)(R13*2), CX
// br3.advance(uint8(v3.entry)
MOVB CH, AL
@@ -667,7 +667,7 @@ bitReader_fillFast_1_end:
MOVQ DI, CX
MOVQ R10, R12
SHRQ CL, R12
MOVW (SI)(R12*2), CX
MOVWQZX (SI)(R12*2), CX
MOVB CH, AL
MOVBQZX CL, CX
ADDQ CX, R11
@@ -675,7 +675,7 @@ bitReader_fillFast_1_end:
MOVQ DI, CX
MOVQ R10, R12
SHRQ CL, R12
MOVW (SI)(R12*2), CX
MOVWQZX (SI)(R12*2), CX
MOVB CH, AH
MOVBQZX CL, CX
ADDQ CX, R11
@@ -694,7 +694,7 @@ bitReader_fillFast_2_end:
MOVQ DI, CX
MOVQ R10, R12
SHRQ CL, R12
MOVW (SI)(R12*2), CX
MOVWQZX (SI)(R12*2), CX
MOVB CH, AH
MOVBQZX CL, CX
ADDQ CX, R11
@@ -702,7 +702,7 @@ bitReader_fillFast_2_end:
MOVQ DI, CX
MOVQ R10, R12
SHRQ CL, R12
MOVW (SI)(R12*2), CX
MOVWQZX (SI)(R12*2), CX
MOVB CH, AL
MOVBQZX CL, CX
ADDQ CX, R11
@@ -769,13 +769,13 @@ main_loop:
bitReader_fillFast_1_end:
SHRXQ DI, R10, CX
MOVW (SI)(CX*2), CX
MOVWQZX (SI)(CX*2), CX
MOVB CH, AL
MOVBQZX CL, CX
ADDQ CX, R11
SHLXQ CX, R10, R10
SHRXQ DI, R10, CX
MOVW (SI)(CX*2), CX
MOVWQZX (SI)(CX*2), CX
MOVB CH, AH
MOVBQZX CL, CX
ADDQ CX, R11
@@ -791,13 +791,13 @@ bitReader_fillFast_1_end:
bitReader_fillFast_2_end:
SHRXQ DI, R10, CX
MOVW (SI)(CX*2), CX
MOVWQZX (SI)(CX*2), CX
MOVB CH, AH
MOVBQZX CL, CX
ADDQ CX, R11
SHLXQ CX, R10, R10
SHRXQ DI, R10, CX
MOVW (SI)(CX*2), CX
MOVWQZX (SI)(CX*2), CX
MOVB CH, AL
MOVBQZX CL, CX
ADDQ CX, R11
vendor/github.com/klauspost/compress/huff0/decompress_arm64.go
Generated Vendored
+37
View File
@@ -0,0 +1,37 @@
//go:build arm64 && !appengine && !noasm && gc
// arm64 stubs and dispatch for the asm loops used by decompress_asm.go.
// The asm (decompress_arm64.s) is generated by the avo arm64 lowering
// printer from the same source as the amd64 asm; see _generate/gen.go.
package huff0
// decompress4x_main_loop_arm64 is an arm64 assembler implementation
// of Decompress4X when tablelog > 8.
//
//go:noescape
func decompress4x_main_loop_arm64(ctx *decompress4xContext)
// decompress4x_8b_main_loop_arm64 is an arm64 assembler implementation
// of Decompress4X when tablelog <= 8 which decodes 4 entries
// per loop.
//
//go:noescape
func decompress4x_8b_main_loop_arm64(ctx *decompress4xContext)
// decompress1x_main_loop_arm64 is an arm64 assembler implementation
// of Decompress1X when tablelog > 8.
//
//go:noescape
func decompress1x_main_loop_arm64(ctx *decompress1xContext)
func decompress4x_main_loop_asm(ctx *decompress4xContext) {
decompress4x_main_loop_arm64(ctx)
}
func decompress4x_8b_main_loop_asm(ctx *decompress4xContext) {
decompress4x_8b_main_loop_arm64(ctx)
}
func decompress1x_main_loop_asm(ctx *decompress1xContext) {
decompress1x_main_loop_arm64(ctx)
}
vendor/github.com/klauspost/compress/huff0/decompress_arm64.s
Generated Vendored
+796
View File
@@ -0,0 +1,796 @@
// Code generated by command: go run gen.go -out ../decompress.s -arch amd64,arm64 -pkg=huff0. DO NOT EDIT.
// EXPERIMENTAL arm64 output lowered from an amd64 avo program.
//go:build arm64 && (!appengine && !noasm && gc)
// func decompress4x_main_loop_amd64(ctx *decompress4xContext)
TEXT ·decompress4x_main_loop_arm64(SB), $0-8
// Preload values
MOVD ctx+0(FP), R0
MOVBU 8(R0), R6
MOVD 16(R0), R3
MOVD 48(R0), R5
MOVD 24(R0), R7
MOVD 32(R0), R8
MOVD (R0), R9
// Main loop
main_loop:
MOVD $0, R2
CMP R5, R3
CSET GE, R16
BFI $0, R16, $8, R2
// br0.fillFast32()
MOVD 32(R9), R10
MOVBU 40(R9), R11
CMP $0x20, R11
BLS skip_fill0
MOVD 24(R9), R0
SUB $0x20, R11, R11
SUB $0x04, R0, R0
MOVD (R9), R12
// b.value |= uint64(low) << (b.bitsRead & 63)
MOVWU (R0)(R12), R12
MOVD R11, R1
LSL R1, R12, R12
MOVD R0, 24(R9)
ORR R12, R10, R10
// exhausted += (br0.off < 4)
CMP $0x04, R0
CSINC HS, R2, R2, R16
BFI $0, R16, $8, R2
skip_fill0:
// val0 := br0.peekTopBits(peekBits)
MOVD R10, R12
MOVD R6, R1
LSR R1, R12, R12
// v0 := table[val0&mask]
MOVHU (R8)(R12<<1), R1
// br0.advance(uint8(v0.entry)
UBFX $8, R1, $8, R0
LSL R1, R10, R10
ADD R1, R11, R15
BFI $0, R15, $8, R11
// val1 := br0.peekTopBits(peekBits)
MOVD R6, R1
MOVD R10, R12
LSR R1, R12, R12
// v1 := table[val1&mask]
MOVHU (R8)(R12<<1), R1
// br0.advance(uint8(v1.entry))
UBFX $8, R1, $8, R16
BFI $8, R16, $8, R0
LSL R1, R10, R10
ADD R1, R11, R15
BFI $0, R15, $8, R11
// these two writes get coalesced
// out[id * dstEvery + 0] = uint8(v0.entry >> 8)
// out[id * dstEvery + 1] = uint8(v1.entry >> 8)
MOVH R0, (R3)
// update the bitreader structure
MOVD R10, 32(R9)
MOVB R11, 40(R9)
// br1.fillFast32()
MOVD 80(R9), R10
MOVBU 88(R9), R11
CMP $0x20, R11
BLS skip_fill1
MOVD 72(R9), R0
SUB $0x20, R11, R11
SUB $0x04, R0, R0
MOVD 48(R9), R12
// b.value |= uint64(low) << (b.bitsRead & 63)
MOVWU (R0)(R12), R12
MOVD R11, R1
LSL R1, R12, R12
MOVD R0, 72(R9)
ORR R12, R10, R10
// exhausted += (br1.off < 4)
CMP $0x04, R0
CSINC HS, R2, R2, R16
BFI $0, R16, $8, R2
skip_fill1:
// val0 := br1.peekTopBits(peekBits)
MOVD R10, R12
MOVD R6, R1
LSR R1, R12, R12
// v0 := table[val0&mask]
MOVHU (R8)(R12<<1), R1
// br1.advance(uint8(v0.entry)
UBFX $8, R1, $8, R0
LSL R1, R10, R10
ADD R1, R11, R15
BFI $0, R15, $8, R11
// val1 := br1.peekTopBits(peekBits)
MOVD R6, R1
MOVD R10, R12
LSR R1, R12, R12
// v1 := table[val1&mask]
MOVHU (R8)(R12<<1), R1
// br1.advance(uint8(v1.entry))
UBFX $8, R1, $8, R16
BFI $8, R16, $8, R0
LSL R1, R10, R10
ADD R1, R11, R15
BFI $0, R15, $8, R11
// these two writes get coalesced
// out[id * dstEvery + 0] = uint8(v0.entry >> 8)
// out[id * dstEvery + 1] = uint8(v1.entry >> 8)
MOVH R0, (R3)(R7)
// update the bitreader structure
MOVD R10, 80(R9)
MOVB R11, 88(R9)
// br2.fillFast32()
MOVD 128(R9), R10
MOVBU 136(R9), R11
CMP $0x20, R11
BLS skip_fill2
MOVD 120(R9), R0
SUB $0x20, R11, R11
SUB $0x04, R0, R0
MOVD 96(R9), R12
// b.value |= uint64(low) << (b.bitsRead & 63)
MOVWU (R0)(R12), R12
MOVD R11, R1
LSL R1, R12, R12
MOVD R0, 120(R9)
ORR R12, R10, R10
// exhausted += (br2.off < 4)
CMP $0x04, R0
CSINC HS, R2, R2, R16
BFI $0, R16, $8, R2
skip_fill2:
// val0 := br2.peekTopBits(peekBits)
MOVD R10, R12
MOVD R6, R1
LSR R1, R12, R12
// v0 := table[val0&mask]
MOVHU (R8)(R12<<1), R1
// br2.advance(uint8(v0.entry)
UBFX $8, R1, $8, R0
LSL R1, R10, R10
ADD R1, R11, R15
BFI $0, R15, $8, R11
// val1 := br2.peekTopBits(peekBits)
MOVD R6, R1
MOVD R10, R12
LSR R1, R12, R12
// v1 := table[val1&mask]
MOVHU (R8)(R12<<1), R1
// br2.advance(uint8(v1.entry))
UBFX $8, R1, $8, R16
BFI $8, R16, $8, R0
LSL R1, R10, R10
ADD R1, R11, R15
BFI $0, R15, $8, R11
// these two writes get coalesced
// out[id * dstEvery + 0] = uint8(v0.entry >> 8)
// out[id * dstEvery + 1] = uint8(v1.entry >> 8)
MOVH R0, (R3)(R7<<1)
// update the bitreader structure
MOVD R10, 128(R9)
MOVB R11, 136(R9)
// br3.fillFast32()
MOVD 176(R9), R10
MOVBU 184(R9), R11
CMP $0x20, R11
BLS skip_fill3
MOVD 168(R9), R0
SUB $0x20, R11, R11
SUB $0x04, R0, R0
MOVD 144(R9), R12
// b.value |= uint64(low) << (b.bitsRead & 63)
MOVWU (R0)(R12), R12
MOVD R11, R1
LSL R1, R12, R12
MOVD R0, 168(R9)
ORR R12, R10, R10
// exhausted += (br3.off < 4)
CMP $0x04, R0
CSINC HS, R2, R2, R16
BFI $0, R16, $8, R2
skip_fill3:
// val0 := br3.peekTopBits(peekBits)
MOVD R10, R12
MOVD R6, R1
LSR R1, R12, R12
// v0 := table[val0&mask]
MOVHU (R8)(R12<<1), R1
// br3.advance(uint8(v0.entry)
UBFX $8, R1, $8, R0
LSL R1, R10, R10
ADD R1, R11, R15
BFI $0, R15, $8, R11
// val1 := br3.peekTopBits(peekBits)
MOVD R6, R1
MOVD R10, R12
LSR R1, R12, R12
// v1 := table[val1&mask]
MOVHU (R8)(R12<<1), R1
// br3.advance(uint8(v1.entry))
UBFX $8, R1, $8, R16
BFI $8, R16, $8, R0
LSL R1, R10, R10
ADD R1, R11, R15
BFI $0, R15, $8, R11
// these two writes get coalesced
// out[id * dstEvery + 0] = uint8(v0.entry >> 8)
// out[id * dstEvery + 1] = uint8(v1.entry >> 8)
ADD R7<<1, R7, R1
MOVH R0, (R3)(R1)
// update the bitreader structure
MOVD R10, 176(R9)
MOVB R11, 184(R9)
ADD $0x02, R3, R3
TST $0xff, R2
BEQ main_loop
MOVD ctx+0(FP), R0
MOVD 16(R0), R16
SUB R16, R3, R3
LSL $0x02, R3, R3
MOVD R3, 40(R0)
RET
// func decompress4x_8b_main_loop_amd64(ctx *decompress4xContext)
TEXT ·decompress4x_8b_main_loop_arm64(SB), $0-8
// Preload values
MOVD ctx+0(FP), R0
MOVBU 8(R0), R6
MOVD 16(R0), R3
MOVD 48(R0), R5
MOVD 24(R0), R7
MOVD 32(R0), R8
MOVD (R0), R9
// Main loop
main_loop:
MOVD $0, R2
CMP R5, R3
CSET GE, R16
BFI $0, R16, $8, R2
// br0.fillFast32()
MOVD 32(R9), R10
MOVBU 40(R9), R11
CMP $0x20, R11
BLS skip_fill0
MOVD 24(R9), R0
SUB $0x20, R11, R11
SUB $0x04, R0, R0
MOVD (R9), R12
// b.value |= uint64(low) << (b.bitsRead & 63)
MOVWU (R0)(R12), R12
MOVD R11, R1
LSL R1, R12, R12
MOVD R0, 24(R9)
ORR R12, R10, R10
// exhausted += (br0.off < 4)
CMP $0x04, R0
CSINC HS, R2, R2, R16
BFI $0, R16, $8, R2
skip_fill0:
// val0 := br0.peekTopBits(peekBits)
MOVD R10, R0
MOVD R6, R1
LSR R1, R0, R0
// v0 := table[val0&mask]
MOVHU (R8)(R0<<1), R1
// br0.advance(uint8(v0.entry)
UBFX $8, R1, $8, R0
LSL R1, R10, R10
ADD R1, R11, R15
BFI $0, R15, $8, R11
// val1 := br0.peekTopBits(peekBits)
MOVD R10, R12
MOVD R6, R1
LSR R1, R12, R12
// v1 := table[val0&mask]
MOVHU (R8)(R12<<1), R1
// br0.advance(uint8(v1.entry)
UBFX $8, R1, $8, R16
BFI $8, R16, $8, R0
LSL R1, R10, R10
ADD R1, R11, R15
BFI $0, R15, $8, R11
REVW R0, R0
// val2 := br0.peekTopBits(peekBits)
MOVD R10, R12
MOVD R6, R1
LSR R1, R12, R12
// v2 := table[val0&mask]
MOVHU (R8)(R12<<1), R1
// br0.advance(uint8(v2.entry)
UBFX $8, R1, $8, R16
BFI $8, R16, $8, R0
LSL R1, R10, R10
ADD R1, R11, R15
BFI $0, R15, $8, R11
// val3 := br0.peekTopBits(peekBits)
MOVD R10, R12
MOVD R6, R1
LSR R1, R12, R12
// v3 := table[val0&mask]
MOVHU (R8)(R12<<1), R1
// br0.advance(uint8(v3.entry)
UBFX $8, R1, $8, R16
BFI $0, R16, $8, R0
LSL R1, R10, R10
ADD R1, R11, R15
BFI $0, R15, $8, R11
REVW R0, R0
// these four writes get coalesced
// out[id * dstEvery + 0] = uint8(v0.entry >> 8)
// out[id * dstEvery + 1] = uint8(v1.entry >> 8)
// out[id * dstEvery + 3] = uint8(v2.entry >> 8)
// out[id * dstEvery + 4] = uint8(v3.entry >> 8)
MOVW R0, (R3)
// update the bitreader structure
MOVD R10, 32(R9)
MOVB R11, 40(R9)
// br1.fillFast32()
MOVD 80(R9), R10
MOVBU 88(R9), R11
CMP $0x20, R11
BLS skip_fill1
MOVD 72(R9), R0
SUB $0x20, R11, R11
SUB $0x04, R0, R0
MOVD 48(R9), R12
// b.value |= uint64(low) << (b.bitsRead & 63)
MOVWU (R0)(R12), R12
MOVD R11, R1
LSL R1, R12, R12
MOVD R0, 72(R9)
ORR R12, R10, R10
// exhausted += (br1.off < 4)
CMP $0x04, R0
CSINC HS, R2, R2, R16
BFI $0, R16, $8, R2
skip_fill1:
// val0 := br1.peekTopBits(peekBits)
MOVD R10, R0
MOVD R6, R1
LSR R1, R0, R0
// v0 := table[val0&mask]
MOVHU (R8)(R0<<1), R1
// br1.advance(uint8(v0.entry)
UBFX $8, R1, $8, R0
LSL R1, R10, R10
ADD R1, R11, R15
BFI $0, R15, $8, R11
// val1 := br1.peekTopBits(peekBits)
MOVD R10, R12
MOVD R6, R1
LSR R1, R12, R12
// v1 := table[val0&mask]
MOVHU (R8)(R12<<1), R1
// br1.advance(uint8(v1.entry)
UBFX $8, R1, $8, R16
BFI $8, R16, $8, R0
LSL R1, R10, R10
ADD R1, R11, R15
BFI $0, R15, $8, R11
REVW R0, R0
// val2 := br1.peekTopBits(peekBits)
MOVD R10, R12
MOVD R6, R1
LSR R1, R12, R12
// v2 := table[val0&mask]
MOVHU (R8)(R12<<1), R1
// br1.advance(uint8(v2.entry)
UBFX $8, R1, $8, R16
BFI $8, R16, $8, R0
LSL R1, R10, R10
ADD R1, R11, R15
BFI $0, R15, $8, R11
// val3 := br1.peekTopBits(peekBits)
MOVD R10, R12
MOVD R6, R1
LSR R1, R12, R12
// v3 := table[val0&mask]
MOVHU (R8)(R12<<1), R1
// br1.advance(uint8(v3.entry)
UBFX $8, R1, $8, R16
BFI $0, R16, $8, R0
LSL R1, R10, R10
ADD R1, R11, R15
BFI $0, R15, $8, R11
REVW R0, R0
// these four writes get coalesced
// out[id * dstEvery + 0] = uint8(v0.entry >> 8)
// out[id * dstEvery + 1] = uint8(v1.entry >> 8)
// out[id * dstEvery + 3] = uint8(v2.entry >> 8)
// out[id * dstEvery + 4] = uint8(v3.entry >> 8)
MOVW R0, (R3)(R7)
// update the bitreader structure
MOVD R10, 80(R9)
MOVB R11, 88(R9)
// br2.fillFast32()
MOVD 128(R9), R10
MOVBU 136(R9), R11
CMP $0x20, R11
BLS skip_fill2
MOVD 120(R9), R0
SUB $0x20, R11, R11
SUB $0x04, R0, R0
MOVD 96(R9), R12
// b.value |= uint64(low) << (b.bitsRead & 63)
MOVWU (R0)(R12), R12
MOVD R11, R1
LSL R1, R12, R12
MOVD R0, 120(R9)
ORR R12, R10, R10
// exhausted += (br2.off < 4)
CMP $0x04, R0
CSINC HS, R2, R2, R16
BFI $0, R16, $8, R2
skip_fill2:
// val0 := br2.peekTopBits(peekBits)
MOVD R10, R0
MOVD R6, R1
LSR R1, R0, R0
// v0 := table[val0&mask]
MOVHU (R8)(R0<<1), R1
// br2.advance(uint8(v0.entry)
UBFX $8, R1, $8, R0
LSL R1, R10, R10
ADD R1, R11, R15
BFI $0, R15, $8, R11
// val1 := br2.peekTopBits(peekBits)
MOVD R10, R12
MOVD R6, R1
LSR R1, R12, R12
// v1 := table[val0&mask]
MOVHU (R8)(R12<<1), R1
// br2.advance(uint8(v1.entry)
UBFX $8, R1, $8, R16
BFI $8, R16, $8, R0
LSL R1, R10, R10
ADD R1, R11, R15
BFI $0, R15, $8, R11
REVW R0, R0
// val2 := br2.peekTopBits(peekBits)
MOVD R10, R12
MOVD R6, R1
LSR R1, R12, R12
// v2 := table[val0&mask]
MOVHU (R8)(R12<<1), R1
// br2.advance(uint8(v2.entry)
UBFX $8, R1, $8, R16
BFI $8, R16, $8, R0
LSL R1, R10, R10
ADD R1, R11, R15
BFI $0, R15, $8, R11
// val3 := br2.peekTopBits(peekBits)
MOVD R10, R12
MOVD R6, R1
LSR R1, R12, R12
// v3 := table[val0&mask]
MOVHU (R8)(R12<<1), R1
// br2.advance(uint8(v3.entry)
UBFX $8, R1, $8, R16
BFI $0, R16, $8, R0
LSL R1, R10, R10
ADD R1, R11, R15
BFI $0, R15, $8, R11
REVW R0, R0
// these four writes get coalesced
// out[id * dstEvery + 0] = uint8(v0.entry >> 8)
// out[id * dstEvery + 1] = uint8(v1.entry >> 8)
// out[id * dstEvery + 3] = uint8(v2.entry >> 8)
// out[id * dstEvery + 4] = uint8(v3.entry >> 8)
ADD R7<<1, R3, R15
MOVW R0, (R15)
// update the bitreader structure
MOVD R10, 128(R9)
MOVB R11, 136(R9)
// br3.fillFast32()
MOVD 176(R9), R10
MOVBU 184(R9), R11
CMP $0x20, R11
BLS skip_fill3
MOVD 168(R9), R0
SUB $0x20, R11, R11
SUB $0x04, R0, R0
MOVD 144(R9), R12
// b.value |= uint64(low) << (b.bitsRead & 63)
MOVWU (R0)(R12), R12
MOVD R11, R1
LSL R1, R12, R12
MOVD R0, 168(R9)
ORR R12, R10, R10
// exhausted += (br3.off < 4)
CMP $0x04, R0
CSINC HS, R2, R2, R16
BFI $0, R16, $8, R2
skip_fill3:
// val0 := br3.peekTopBits(peekBits)
MOVD R10, R0
MOVD R6, R1
LSR R1, R0, R0
// v0 := table[val0&mask]
MOVHU (R8)(R0<<1), R1
// br3.advance(uint8(v0.entry)
UBFX $8, R1, $8, R0
LSL R1, R10, R10
ADD R1, R11, R15
BFI $0, R15, $8, R11
// val1 := br3.peekTopBits(peekBits)
MOVD R10, R12
MOVD R6, R1
LSR R1, R12, R12
// v1 := table[val0&mask]
MOVHU (R8)(R12<<1), R1
// br3.advance(uint8(v1.entry)
UBFX $8, R1, $8, R16
BFI $8, R16, $8, R0
LSL R1, R10, R10
ADD R1, R11, R15
BFI $0, R15, $8, R11
REVW R0, R0
// val2 := br3.peekTopBits(peekBits)
MOVD R10, R12
MOVD R6, R1
LSR R1, R12, R12
// v2 := table[val0&mask]
MOVHU (R8)(R12<<1), R1
// br3.advance(uint8(v2.entry)
UBFX $8, R1, $8, R16
BFI $8, R16, $8, R0
LSL R1, R10, R10
ADD R1, R11, R15
BFI $0, R15, $8, R11
// val3 := br3.peekTopBits(peekBits)
MOVD R10, R12
MOVD R6, R1
LSR R1, R12, R12
// v3 := table[val0&mask]
MOVHU (R8)(R12<<1), R1
// br3.advance(uint8(v3.entry)
UBFX $8, R1, $8, R16
BFI $0, R16, $8, R0
LSL R1, R10, R10
ADD R1, R11, R15
BFI $0, R15, $8, R11
REVW R0, R0
// these four writes get coalesced
// out[id * dstEvery + 0] = uint8(v0.entry >> 8)
// out[id * dstEvery + 1] = uint8(v1.entry >> 8)
// out[id * dstEvery + 3] = uint8(v2.entry >> 8)
// out[id * dstEvery + 4] = uint8(v3.entry >> 8)
ADD R7<<1, R7, R1
MOVW R0, (R3)(R1)
// update the bitreader structure
MOVD R10, 176(R9)
MOVB R11, 184(R9)
ADD $0x04, R3, R3
TST $0xff, R2
BEQ main_loop
MOVD ctx+0(FP), R0
MOVD 16(R0), R16
SUB R16, R3, R3
LSL $0x02, R3, R3
MOVD R3, 40(R0)
RET
// func decompress1x_main_loop_amd64(ctx *decompress1xContext)
TEXT ·decompress1x_main_loop_arm64(SB), $0-8
MOVD ctx+0(FP), R1
MOVD 16(R1), R2
MOVD 24(R1), R3
CMP $0x04, R3
BLO error_max_decoded_size_exceeded
ADD R3, R2, R3
MOVD (R1), R5
MOVD (R5), R7
MOVD 24(R5), R8
MOVD 32(R5), R9
MOVBU 40(R5), R10
MOVD 32(R1), R5
MOVBU 8(R1), R6
JMP loop_condition
main_loop:
// Check if we have room for 4 bytes in the output buffer
ADD $4, R2, R1
CMP R3, R1
BGE error_max_decoded_size_exceeded
// Decode 4 values
CMP $0x20, R10
BLT bitReader_fillFast_1_end
SUB $0x20, R10, R10
SUB $0x04, R8, R8
MOVWU (R7)(R8), R11
MOVD R10, R1
LSL R1, R11, R11
ORR R11, R9, R9
bitReader_fillFast_1_end:
MOVD R6, R1
MOVD R9, R11
LSR R1, R11, R11
MOVHU (R5)(R11<<1), R1
UBFX $8, R1, $8, R16
BFI $0, R16, $8, R0
MOVBU R1, R1
ADD R1, R10, R10
LSL R1, R9, R9
MOVD R6, R1
MOVD R9, R11
LSR R1, R11, R11
MOVHU (R5)(R11<<1), R1
UBFX $8, R1, $8, R16
BFI $8, R16, $8, R0
MOVBU R1, R1
ADD R1, R10, R10
LSL R1, R9, R9
REVW R0, R0
CMP $0x20, R10
BLT bitReader_fillFast_2_end
SUB $0x20, R10, R10
SUB $0x04, R8, R8
MOVWU (R7)(R8), R11
MOVD R10, R1
LSL R1, R11, R11
ORR R11, R9, R9
bitReader_fillFast_2_end:
MOVD R6, R1
MOVD R9, R11
LSR R1, R11, R11
MOVHU (R5)(R11<<1), R1
UBFX $8, R1, $8, R16
BFI $8, R16, $8, R0
MOVBU R1, R1
ADD R1, R10, R10
LSL R1, R9, R9
MOVD R6, R1
MOVD R9, R11
LSR R1, R11, R11
MOVHU (R5)(R11<<1), R1
UBFX $8, R1, $8, R16
BFI $0, R16, $8, R0
MOVBU R1, R1
ADD R1, R10, R10
LSL R1, R9, R9
REVW R0, R0
// Store the decoded values
MOVW R0, (R2)
ADD $0x04, R2, R2
loop_condition:
CMP $0x08, R8
BGE main_loop
// Update ctx structure
MOVD ctx+0(FP), R0
MOVD 16(R0), R16
SUB R16, R2, R2
MOVD R2, 40(R0)
MOVD (R0), R0
MOVD R8, 24(R0)
MOVD R9, 32(R0)
MOVB R10, 40(R0)
RET
// Report error
error_max_decoded_size_exceeded:
MOVD ctx+0(FP), R0
MOVD $-1, R1
MOVD R1, 40(R0)
RET
// skipped decompress1x_main_loop_bmi2 (generic twin preferred on arm64)
vendor/github.com/klauspost/compress/huff0/decompress_asm.go
Generated Vendored
+193
View File
@@ -0,0 +1,193 @@
//go:build (amd64 || arm64) && !appengine && !noasm && gc
// This file contains the specialisation of Decoder.Decompress4X
// and Decoder.Decompress1X that use an asm implementation of their main loops.
// The asm function stubs and any per-arch dispatch live in decompress_amd64.go
// and decompress_arm64.go.
package huff0
import (
"errors"
"fmt"
)
// fallback8BitSize is the size where using Go version is faster.
const fallback8BitSize = 800
type decompress4xContext struct {
pbr *[4]bitReaderShifted
peekBits uint8
out *byte
dstEvery int
tbl *dEntrySingle
decoded int
limit *byte
}
// Decompress4X will decompress a 4X encoded stream.
// The length of the supplied input must match the end of a block exactly.
// The *capacity* of the dst slice must match the destination size of
// the uncompressed data exactly.
func (d *Decoder) Decompress4X(dst, src []byte) ([]byte, error) {
if len(d.dt.single) == 0 {
return nil, errors.New("no table loaded")
}
if len(src) < 6+(4*1) {
return nil, errors.New("input too small")
}
use8BitTables := d.actualTableLog <= 8
if cap(dst) < fallback8BitSize && use8BitTables {
return d.decompress4X8bit(dst, src)
}
var br [4]bitReaderShifted
// Decode "jump table"
start := 6
for i := range 3 {
length := int(src[i*2]) | (int(src[i*2+1]) << 8)
if start+length >= len(src) {
return nil, errors.New("truncated input (or invalid offset)")
}
err := br[i].init(src[start : start+length])
if err != nil {
return nil, err
}
start += length
}
err := br[3].init(src[start:])
if err != nil {
return nil, err
}
// destination, offset to match first output
dstSize := cap(dst)
dst = dst[:dstSize]
out := dst
dstEvery := (dstSize + 3) / 4
const tlSize = 1 << tableLogMax
const tlMask = tlSize - 1
single := d.dt.single[:tlSize]
var decoded int
if len(out) > 4*4 && !(br[0].off < 4 || br[1].off < 4 || br[2].off < 4 || br[3].off < 4) {
ctx := decompress4xContext{
pbr: &br,
peekBits: uint8((64 - d.actualTableLog) & 63), // see: bitReaderShifted.peekBitsFast()
out: &out[0],
dstEvery: dstEvery,
tbl: &single[0],
limit: &out[dstEvery-4], // Always stop decoding when first buffer gets here to avoid writing OOB on last.
}
if use8BitTables {
decompress4x_8b_main_loop_asm(&ctx)
} else {
decompress4x_main_loop_asm(&ctx)
}
decoded = ctx.decoded
out = out[decoded/4:]
}
// Decode remaining.
remainBytes := dstEvery - (decoded / 4)
for i := range br {
offset := dstEvery * i
endsAt := min(offset+remainBytes, len(out))
br := &br[i]
bitsLeft := br.remaining()
for bitsLeft > 0 {
br.fill()
if offset >= endsAt {
return nil, errors.New("corruption detected: stream overrun 4")
}
// Read value and increment offset.
val := br.peekBitsFast(d.actualTableLog)
v := single[val&tlMask].entry
nBits := uint8(v)
br.advance(nBits)
bitsLeft -= uint(nBits)
out[offset] = uint8(v >> 8)
offset++
}
if offset != endsAt {
return nil, fmt.Errorf("corruption detected: short output block %d, end %d != %d", i, offset, endsAt)
}
decoded += offset - dstEvery*i
err = br.close()
if err != nil {
return nil, err
}
}
if dstSize != decoded {
return nil, errors.New("corruption detected: short output block")
}
return dst, nil
}
type decompress1xContext struct {
pbr *bitReaderShifted
peekBits uint8
out *byte
outCap int
tbl *dEntrySingle
decoded int
}
// Error reported by asm implementations
const error_max_decoded_size_exeeded = -1
// Decompress1X will decompress a 1X encoded stream.
// The cap of the output buffer will be the maximum decompressed size.
// The length of the supplied input must match the end of a block exactly.
func (d *Decoder) Decompress1X(dst, src []byte) ([]byte, error) {
if len(d.dt.single) == 0 {
return nil, errors.New("no table loaded")
}
var br bitReaderShifted
err := br.init(src)
if err != nil {
return dst, err
}
maxDecodedSize := cap(dst)
dst = dst[:maxDecodedSize]
const tlSize = 1 << tableLogMax
const tlMask = tlSize - 1
if maxDecodedSize >= 4 {
ctx := decompress1xContext{
pbr: &br,
out: &dst[0],
outCap: maxDecodedSize,
peekBits: uint8((64 - d.actualTableLog) & 63), // see: bitReaderShifted.peekBitsFast()
tbl: &d.dt.single[0],
}
decompress1x_main_loop_asm(&ctx)
if ctx.decoded == error_max_decoded_size_exeeded {
return nil, ErrMaxDecodedSizeExceeded
}
dst = dst[:ctx.decoded]
}
// br < 8, so uint8 is fine
bitsLeft := uint8(br.off)*8 + 64 - br.bitsRead
for bitsLeft > 0 {
br.fill()
if len(dst) >= maxDecodedSize {
br.close()
return nil, ErrMaxDecodedSizeExceeded
}
v := d.dt.single[br.peekBitsFast(d.actualTableLog)&tlMask]
nBits := uint8(v.entry)
br.advance(nBits)
bitsLeft -= nBits
dst = append(dst, uint8(v.entry>>8))
}
return dst, br.close()
}
vendor/github.com/klauspost/compress/huff0/decompress_generic.go
Generated Vendored
+1 -2
View File
@@ -1,5 +1,4 @@
//go:build !amd64 || appengine || !gc || noasm
// +build !amd64 appengine !gc noasm
//go:build (!amd64 && !arm64) || appengine || !gc || noasm
// This file contains a generic implementation of Decoder.Decompress4X.
package huff0