Skip to content

Commit 71295ed

Browse files
author
Timothy B. Terriberry
committed
Add Daala's entropy coder to Thor.
Daala's entropy coder supports a "raw bits" mode which just writes bits directly, working backwards from the end of the stream. This should be essentially as fast as Thor's current bit writing and reading code. This patch replaces that code with Daala's, allowing future patches to use the non-raw entropy coder. This lets the existing code be converted a piece at a time. Unfortuntaely, because Thor uses VLCs and does not match the number of bits written at the encoder with the number of bits it attempts to read at the decoder, the order the bits are written to the stream matters. As a short-term solution, we currently bitreverse all bits before writing and bitreverse them again after reading, so that the relative order of bits remains the same. It would also be possible to modify the calling code to expect the bits to be in a different order without any of this overhead.
1 parent 4a78924 commit 71295ed

File tree

17 files changed

+2181
-287
lines changed

17 files changed

+2181
-287
lines changed

‎Makefile‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ endif
2020
COMMON_SOURCES = \
2121
common/common_block.c \
2222
common/common_frame.c \
23+
common/entcode.c \
2324
common/transform.c \
2425
common/intra_prediction.c \
2526
common/inter_prediction.c \
@@ -31,6 +32,7 @@ COMMON_SOURCES = \
3132
ENCODER_SOURCES = \
3233
enc/encode_block.c \
3334
enc/encode_frame.c \
35+
enc/entenc.c \
3436
enc/mainenc.c \
3537
enc/putbits.c \
3638
enc/putvlc.c \
@@ -41,6 +43,7 @@ ENCODER_SOURCES = \
4143

4244
DECODER_SOURCES = \
4345
dec/decode_block.c \
46+
dec/entdec.c \
4447
dec/getbits.c \
4548
dec/getvlc.c \
4649
dec/maindec.c \

‎common/entcode.c‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
/*Daala video codec
2+
Copyright (c) 2001-2012 Daala project contributors. All rights reserved.
3+
4+
Redistribution and use in source and binary forms, with or without
5+
modification, are permitted provided that the following conditions are met:
6+
7+
- Redistributions of source code must retain the above copyright notice, this
8+
list of conditions and the following disclaimer.
9+
10+
- Redistributions in binary form must reproduce the above copyright notice,
11+
this list of conditions and the following disclaimer in the documentation
12+
and/or other materials provided with the distribution.
13+
14+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
15+
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
16+
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
17+
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE
18+
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
19+
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
20+
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
21+
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
22+
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
23+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.*/
24+
25+
#ifdef HAVE_CONFIG_H
26+
# include "config.h"
27+
#endif
28+
29+
#include "entcode.h"
30+
31+
/*CDFs for uniform probability distributions of small sizes (2 through 16,
32+
inclusive).*/
33+
const uint16_t OD_UNIFORM_CDFS_Q15[135] = {
34+
16384, 32768,
35+
10923, 21845, 32768,
36+
8192, 16384, 24576, 32768,
37+
6554, 13107, 19661, 26214, 32768,
38+
5461, 10923, 16384, 21845, 27307, 32768,
39+
4681, 9362, 14043, 18725, 23406, 28087, 32768,
40+
4096, 8192, 12288, 16384, 20480, 24576, 28672, 32768,
41+
3641, 7282, 10923, 14564, 18204, 21845, 25486, 29127, 32768,
42+
3277, 6554, 9830, 13107, 16384, 19661, 22938, 26214, 29491, 32768,
43+
2979, 5958, 8937, 11916, 14895, 17873, 20852, 23831, 26810, 29789, 32768,
44+
2731, 5461, 8192, 10923, 13653, 16384, 19115, 21845, 24576, 27307, 30037,
45+
32768,
46+
2521, 5041, 7562, 10082, 12603, 15124, 17644, 20165, 22686, 25206, 27727,
47+
30247, 32768,
48+
2341, 4681, 7022, 9362, 11703, 14043, 16384, 18725, 21065, 23406, 25746,
49+
28087, 30427, 32768,
50+
2185, 4369, 6554, 8738, 10923, 13107, 15292, 17476, 19661, 21845, 24030,
51+
26214, 28399, 30583, 32768,
52+
2048, 4096, 6144, 8192, 10240, 12288, 14336, 16384, 18432, 20480, 22528,
53+
24576, 26624, 28672, 30720, 32768
54+
};
55+
56+
/*Given the current total integer number of bits used and the current value of
57+
rng, computes the fraction number of bits used to OD_BITRES precision.
58+
This is used by od_ec_enc_tell_frac() and od_ec_dec_tell_frac().
59+
nbits_total: The number of whole bits currently used, i.e., the value
60+
returned by od_ec_enc_tell() or od_ec_dec_tell().
61+
rng: The current value of rng from either the encoder or decoder state.
62+
Return: The number of bits scaled by 2**OD_BITRES.
63+
This will always be slightly larger than the exact value (e.g., all
64+
rounding error is in the positive direction).*/
65+
uint32_t od_ec_tell_frac(uint32_t nbits_total, uint32_t rng) {
66+
uint32_t nbits;
67+
int l;
68+
int i;
69+
/*To handle the non-integral number of bits still left in the encoder/decoder
70+
state, we compute the worst-case number of bits of val that must be
71+
encoded to ensure that the value is inside the range for any possible
72+
subsequent bits.
73+
The computation here is independent of val itself (the decoder does not
74+
even track that value), even though the real number of bits used after
75+
od_ec_enc_done() may be 1 smaller if rng is a power of two and the
76+
corresponding trailing bits of val are all zeros.
77+
If we did try to track that special case, then coding a value with a
78+
probability of 1/(1 << n) might sometimes appear to use more than n bits.
79+
This may help explain the surprising result that a newly initialized
80+
encoder or decoder claims to have used 1 bit.*/
81+
nbits = nbits_total << OD_BITRES;
82+
l = 0;
83+
for (i = OD_BITRES; i-- > 0;) {
84+
int b;
85+
rng = rng*rng >> 15;
86+
b = (int)(rng >> 16);
87+
l = l << 1 | b;
88+
rng >>= b;
89+
}
90+
return nbits - l;
91+
}

‎common/entcode.h‎

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
/*Daala video codec
2+
Copyright (c) 2001-2013 Daala project contributors. All rights reserved.
3+
4+
Redistribution and use in source and binary forms, with or without
5+
modification, are permitted provided that the following conditions are met:
6+
7+
- Redistributions of source code must retain the above copyright notice, this
8+
list of conditions and the following disclaimer.
9+
10+
- Redistributions in binary form must reproduce the above copyright notice,
11+
this list of conditions and the following disclaimer in the documentation
12+
and/or other materials provided with the distribution.
13+
14+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
15+
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
16+
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
17+
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE
18+
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
19+
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
20+
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
21+
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
22+
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
23+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.*/
24+
25+
#if !defined(_entcode_H)
26+
# define _entcode_H (1)
27+
# include <limits.h>
28+
# include <stddef.h>
29+
# include <stdint.h>
30+
# include "odintrin.h"
31+
32+
/*Set this flag 1 to enable a "reduced overhead" version of the entropy coder.
33+
This uses a partition function that more accurately follows the input
34+
probability estimates at the expense of some additional CPU cost (though
35+
still an order of magnitude less than a full division).
36+
37+
In classic arithmetic coding, the partition function maps a value x in the
38+
range [0, ft] to a value in y in [0, r] with 0 < ft <= r via
39+
y = x*r/ft.
40+
Any deviation from this value increases coding inefficiency.
41+
42+
To avoid divisions, we require ft <= r < 2*ft (enforcing it by shifting up
43+
ft if necessary), and replace that function with
44+
y = x + OD_MINI(x, r - ft).
45+
This counts values of x smaller than r - ft double compared to values larger
46+
than r - ft, which over-estimates the probability of symbols at the start of
47+
the alphabet, and under-estimates the probability of symbols at the end of
48+
the alphabet.
49+
The overall coding inefficiency assuming accurate probability models and
50+
independent symbols is in the 1% range, which is similar to that of CABAC.
51+
52+
To reduce overhead even further, we split this into two cases:
53+
1) r - ft > ft - (r - ft).
54+
That is, we have more values of x that are double-counted than
55+
single-counted.
56+
In this case, we still double-count the first 2*r - 3*ft values of x, but
57+
after that we alternate between single-counting and double-counting for
58+
the rest.
59+
2) r - ft < ft - (r - ft).
60+
That is, we have more values of x that are single-counted than
61+
double-counted.
62+
In this case, we alternate between single-counting and double-counting for
63+
the first 2*(r - ft) values of x, and single-count the rest.
64+
For two equiprobable symbols in different places in the alphabet, this
65+
reduces the maximum ratio of over-estimation to under-estimation from 2:1
66+
for the previous partition function to either 4:3 or 3:2 (for each of the
67+
two cases above, respectively), assuming symbol probabilities significantly
68+
greater than 1/32768.
69+
That reduces the worst-case per-symbol overhead from 1 bit to 0.58 bits.
70+
71+
The resulting function is
72+
e = OD_MAXI(2*r - 3*ft, 0);
73+
y = x + OD_MINI(x, e) + OD_MINI(OD_MAXI(x - e, 0) >> 1, r - ft).
74+
Here, e is a value that is greater than 0 in case 1, and 0 in case 2.
75+
This function is about 3 times as expensive to evaluate as the high-overhead
76+
version, but still an order of magnitude cheaper than a division, since it
77+
is composed only of very simple operations.
78+
Because we want to fit in 16-bit registers and must use unsigned values to do
79+
so, we use saturating subtraction to enforce the maximums with 0.
80+
81+
Enabling this reduces the measured overhead in ectest from 0.805% to 0.621%
82+
(vs. 0.022% for the division-based partition function with r much greater
83+
than ft).
84+
It improves performance on ntt-short-1 by about 0.3%.*/
85+
# define OD_EC_REDUCED_OVERHEAD (0)
86+
87+
/*OPT: od_ec_window must be at least 32 bits, but if you have fast arithmetic
88+
on a larger type, you can speed up the decoder by using it here.*/
89+
typedef uint32_t od_ec_window;
90+
91+
# define OD_EC_WINDOW_SIZE ((int)sizeof(od_ec_window)*CHAR_BIT)
92+
93+
/*Unsigned subtraction with unsigned saturation.
94+
This implementation of the macro is intentionally chosen to increase the
95+
number of common subexpressions in the reduced-overhead partition function.
96+
This matters for C code, but it would not for hardware with a saturating
97+
subtraction instruction.*/
98+
#define OD_SUBSATU(a, b) ((a) - OD_MINI(a, b))
99+
100+
/*The number of bits to use for the range-coded part of unsigned integers.*/
101+
# define OD_EC_UINT_BITS (4)
102+
103+
/*The resolution of fractional-precision bit usage measurements, i.e.,
104+
3 => 1/8th bits.*/
105+
# define OD_BITRES (3)
106+
107+
extern const uint16_t OD_UNIFORM_CDFS_Q15[135];
108+
109+
/*Returns a Q15 CDF for a uniform probability distribution of the given size.
110+
n: The size of the distribution.
111+
This must be at least 2, and no more than 16.*/
112+
# define OD_UNIFORM_CDF_Q15(n) \
113+
(OD_UNIFORM_CDFS_Q15 + ((n)*((n) - 1) >> 1) - 1)
114+
115+
/*See entcode.c for further documentation.*/
116+
117+
OD_WARN_UNUSED_RESULT uint32_t od_ec_tell_frac(uint32_t nbits_total,
118+
uint32_t rng);
119+
120+
#endif

0 commit comments

Comments
 (0)