14736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh/* 24736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * Copyright (C) 2010 The Android Open Source Project 34736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * 44736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * Licensed under the Apache License, Version 2.0 (the "License"); 54736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * you may not use this file except in compliance with the License. 64736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * You may obtain a copy of the License at 74736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * 84736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * http://www.apache.org/licenses/LICENSE-2.0 94736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * 104736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * Unless required by applicable law or agreed to in writing, software 114736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * distributed under the License is distributed on an "AS IS" BASIS, 124736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 134736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * See the License for the specific language governing permissions and 144736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * limitations under the License. 154736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh */ 164736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 174736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh/* 184736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * This is a fast-and-accurate implementation of inverse Discrete Cosine 194736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * Transform (IDCT) for ARMv6+. It also performs dequantization of the input 204736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * coefficients just like other methods. 214736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * 224736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * This implementation is based on the scaled 1-D DCT algorithm proposed by 234736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * Arai, Agui, and Nakajima. The following code is based on the figure 4-8 244736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * on page 52 of the JPEG textbook by Pennebaker and Mitchell. Coefficients 254736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * are (almost) directly mapped into registers. 264736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * 274736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * The accuracy is achieved by using SMULWy and SMLAWy instructions. Both 284736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * multiply 32 bits by 16 bits and store the top 32 bits of the result. It 294736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * makes 32-bit fixed-point arithmetic possible without overflow. That is 304736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * why jpeg_idct_ifast(), which is written in C, cannot be improved. 314736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * 324736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * More tricks are used to gain more speed. First of all, we use as many 334736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * registers as possible. ARM processor has 16 registers including sp (r13) 344736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * and pc (r15), so only 14 registers can be used without limitations. In 354736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * general, we let r0 to r7 hold the coefficients; r10 and r11 hold four 364736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * 16-bit constants; r12 and r14 hold two of the four arguments; and r8 hold 374736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * intermediate value. In the second pass, r9 is the loop counter. In the 384736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * first pass, r8 to r11 are used to hold quantization values, so the loop 394736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * counter is held by sp. Yes, the stack pointer. Since it must be aligned 404736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * to 4-byte boundary all the time, we align it to 32-byte boundary and use 414736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * bit 3 to bit 5. As the result, we actually use 14.1 registers. :-) 424736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * 434736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * Second, we rearrange quantization values to access them sequentially. The 444736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * table is first transposed, and the new columns are placed in the order of 454736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * 7, 5, 1, 3, 0, 2, 4, 6. Thus we can use LDMDB to load four values at a 464736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * time. Rearranging coefficients also helps, but that requires to change a 474736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * dozen of files, which seems not worth it. In addition, we choose to scale 484736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * up quantization values by 13 bits, so the coefficients are scaled up by 494736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * 16 bits after both passes. Then we can pack and saturate them two at a 504736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * time using PKHTB and USAT16 instructions. 514736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * 524736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * Third, we reorder the instructions to avoid bubbles in the pipeline. This 534736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * is done by hand accroding to the cycle timings and the interlock behavior 544736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * described in the technical reference manual of ARM1136JF-S. We also take 554736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * advantage of dual issue processors by interleaving instructions with 564736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * dependencies. It has been benchmarked on four devices and all the results 574736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * showed distinguishable improvements. Note that PLD instructions actually 584736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * slow things down, so they are removed at the last minute. In the future, 594736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh * this might be futher improved using a system profiler. 604736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh */ 614736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 624736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh#ifdef __arm__ 634736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh#include <machine/cpu-features.h> 644736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh#endif 654736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 664736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh#if __ARM_ARCH__ >= 6 674736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 684736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh// void armv6_idct(short *coefs, int *quans, unsigned char *rows, int col) 694736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh .arm 704736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh .text 714736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh .align 724736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh .global armv6_idct 734736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh .func armv6_idct 744736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 754736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeharmv6_idct: 764736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // Push everything except sp (r13) and pc (r15). 774736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh stmdb sp!, {r4, r5, r6, r7, r8, r9, r10, r11, r12, r14} 784736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 794736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r12 = quans, r14 = coefs. 804736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh sub r4, sp, #236 814736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh bic sp, r4, #31 824736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh add r5, sp, #224 834736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh add r12, r1, #256 844736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh stm r5, {r2, r3, r4} 854736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh add r14, r0, #16 864736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 874736a38e2514bfe50b0241d2053befea82822944Chia-chi Yehpass1_head: 884736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // Load quantization values. (q[0, 2, 4, 6]) 894736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh ldmdb r12!, {r8, r9, r10, r11} 904736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 914736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // Load coefficients. (c[4, 1, 2, 3, 0, 5, 6, 7]) 924736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh ldrsh r4, [r14, #-2] ! 934736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh ldrsh r1, [r14, #16] 944736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh ldrsh r2, [r14, #32] 954736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh ldrsh r3, [r14, #48] 964736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh ldrsh r0, [r14, #64] 974736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh ldrsh r5, [r14, #80] 984736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh ldrsh r6, [r14, #96] 994736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh ldrsh r7, [r14, #112] 1004736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 1014736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r4 = q[0] * c[0]; 1024736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh mul r4, r8, r4 1034736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 1044736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // Check if ACs are all zero. 1054736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh cmp r0, #0 1064736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh orreqs r8, r1, r2 1074736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh orreqs r8, r3, r5 1084736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh orreqs r8, r6, r7 1094736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh beq pass1_zero 1104736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 1114736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // Step 1: Dequantizations. 1124736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 1134736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r2 = q[2] * c[2]; 1144736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r0 = q[4] * c[4] + r4; 1154736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r6 = q[6] * c[6] + r2; 1164736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh mul r2, r9, r2 1174736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh mla r0, r10, r0, r4 1184736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh mla r6, r11, r6, r2 1194736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 1204736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // Load quantization values. (q[7, 5, 1, 3]) 1214736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh ldmdb r12!, {r8, r9, r10, r11} 1224736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 1234736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r4 = r4 * 2 - r0 = -(r0 - r4 * 2); 1244736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r2 = r2 * 2 - r6 = -(r6 - r2 * 2); 1254736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh rsb r4, r0, r4, lsl #1 1264736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh rsb r2, r6, r2, lsl #1 1274736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 1284736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r7 = q[7] * c[7]; 1294736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r5 = q[5] * c[5]; 1304736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r1 = q[1] * c[1] + r7; 1314736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r3 = q[3] * c[3] + r5; 1324736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh mul r7, r8, r7 1334736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh mul r5, r9, r5 1344736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh mla r1, r10, r1, r7 1354736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh mla r3, r11, r3, r5 1364736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 1374736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // Load constants. 1384736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh ldrd r10, constants 1394736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 1404736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // Step 2: Rotations and Butterflies. 1414736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 1424736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r7 = r1 - r7 * 2; 1434736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r1 = r1 - r3; 1444736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r5 = r5 * 2 - r3 = -(r3 - r5 * 2); 1454736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r3 = r1 + r3 * 2; 1464736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r8 = r5 + r7; 1474736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh sub r7, r1, r7, lsl #1 1484736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh sub r1, r1, r3 1494736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh rsb r5, r3, r5, lsl #1 1504736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh add r3, r1, r3, lsl #1 1514736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh add r8, r5, r7 1524736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 1534736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r2 = r2 * 1.41421 = r2 * 27146 / 65536 + r2; 1544736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r8 = r8 * 1.84776 / 8 = r8 * 15137 / 65536; 1554736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r1 = r1 * 1.41421 = r1 * 27146 / 65536 + r1; 1564736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh smlawt r2, r2, r10, r2 1574736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh smulwb r8, r8, r10 1584736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh smlawt r1, r1, r10, r1 1594736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 1604736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r0 = r0 + r6; 1614736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r2 = r2 - r6; 1624736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r6 = r0 - r6 * 2; 1634736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh add r0, r0, r6 1644736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh sub r2, r2, r6 1654736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh sub r6, r0, r6, lsl #1 1664736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 1674736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r5 = r5 * -2.61313 / 8 + r8 = r5 * -21407 / 65536 + r8; 1684736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r8 = r7 * -1.08239 / 8 + r8 = r7 * -8867 / 65536 + r8; 1694736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh smlawt r5, r5, r11, r8 1704736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh smlawb r8, r7, r11, r8 1714736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 1724736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r4 = r4 + r2; 1734736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r0 = r0 + r3; 1744736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r2 = r4 - r2 * 2; 1754736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh add r4, r4, r2 1764736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh add r0, r0, r3 1774736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh sub r2, r4, r2, lsl #1 1784736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 1794736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r7 = r5 * 8 - r3 = -(r3 - r5 * 8); 1804736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r3 = r0 - r3 * 2; 1814736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r1 = r1 - r7; 1824736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r4 = r4 + r7; 1834736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r5 = r8 * 8 - r1 = -(r1 - r8 * 8); 1844736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r7 = r4 - r7 * 2; 1854736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh rsb r7, r3, r5, lsl #3 1864736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh sub r3, r0, r3, lsl #1 1874736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh sub r1, r1, r7 1884736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh add r4, r4, r7 1894736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh rsb r5, r1, r8, lsl #3 1904736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh sub r7, r4, r7, lsl #1 1914736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 1924736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r2 = r2 + r1; 1934736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r6 = r6 + r5; 1944736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r1 = r2 - r1 * 2; 1954736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r5 = r6 - r5 * 2; 1964736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh add r2, r2, r1 1974736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh add r6, r6, r5 1984736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh sub r1, r2, r1, lsl #1 1994736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh sub r5, r6, r5, lsl #1 2004736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 2014736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // Step 3: Reorder and Save. 2024736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 2034736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh str r0, [sp, #-4] ! 2044736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh str r4, [sp, #32] 2054736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh str r2, [sp, #64] 2064736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh str r6, [sp, #96] 2074736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh str r5, [sp, #128] 2084736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh str r1, [sp, #160] 2094736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh str r7, [sp, #192] 2104736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh str r3, [sp, #224] 2114736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh b pass1_tail 2124736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 2134736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // Precomputed 16-bit constants: 27146, 15137, -21407, -8867. 2144736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // Put them in the middle since LDRD only accepts offsets from -255 to 255. 2154736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh .align 3 2164736a38e2514bfe50b0241d2053befea82822944Chia-chi Yehconstants: 2174736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh .word 0x6a0a3b21 2184736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh .word 0xac61dd5d 2194736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 2204736a38e2514bfe50b0241d2053befea82822944Chia-chi Yehpass1_zero: 2214736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh str r4, [sp, #-4] ! 2224736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh str r4, [sp, #32] 2234736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh str r4, [sp, #64] 2244736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh str r4, [sp, #96] 2254736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh str r4, [sp, #128] 2264736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh str r4, [sp, #160] 2274736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh str r4, [sp, #192] 2284736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh str r4, [sp, #224] 2294736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh sub r12, r12, #16 2304736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 2314736a38e2514bfe50b0241d2053befea82822944Chia-chi Yehpass1_tail: 2324736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh ands r9, sp, #31 2334736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh bne pass1_head 2344736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 2354736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r12 = rows, r14 = col. 2364736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh ldr r12, [sp, #256] 2374736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh ldr r14, [sp, #260] 2384736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 2394736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // Load constants. 2404736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh ldrd r10, constants 2414736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 2424736a38e2514bfe50b0241d2053befea82822944Chia-chi Yehpass2_head: 2434736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // Load coefficients. (c[0, 1, 2, 3, 4, 5, 6, 7]) 2444736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh ldmia sp!, {r0, r1, r2, r3, r4, r5, r6, r7} 2454736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 2464736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r0 = r0 + 0x00808000; 2474736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh add r0, r0, #0x00800000 2484736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh add r0, r0, #0x00008000 2494736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 2504736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // Step 1: Analog to the first pass. 2514736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 2524736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r0 = r0 + r4; 2534736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r6 = r6 + r2; 2544736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh add r0, r0, r4 2554736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh add r6, r6, r2 2564736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 2574736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r4 = r0 - r4 * 2; 2584736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r2 = r2 * 2 - r6 = -(r6 - r2 * 2); 2594736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh sub r4, r0, r4, lsl #1 2604736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh rsb r2, r6, r2, lsl #1 2614736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 2624736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r1 = r1 + r7; 2634736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r3 = r3 + r5; 2644736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh add r1, r1, r7 2654736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh add r3, r3, r5 2664736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 2674736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // Step 2: Rotations and Butterflies. 2684736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 2694736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r7 = r1 - r7 * 2; 2704736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r1 = r1 - r3; 2714736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r5 = r5 * 2 - r3 = -(r3 - r5 * 2); 2724736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r3 = r1 + r3 * 2; 2734736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r8 = r5 + r7; 2744736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh sub r7, r1, r7, lsl #1 2754736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh sub r1, r1, r3 2764736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh rsb r5, r3, r5, lsl #1 2774736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh add r3, r1, r3, lsl #1 2784736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh add r8, r5, r7 2794736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 2804736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r2 = r2 * 1.41421 = r2 * 27146 / 65536 + r2; 2814736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r8 = r8 * 1.84776 / 8 = r8 * 15137 / 65536; 2824736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r1 = r1 * 1.41421 = r1 * 27146 / 65536 + r1; 2834736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh smlawt r2, r2, r10, r2 2844736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh smulwb r8, r8, r10 2854736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh smlawt r1, r1, r10, r1 2864736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 2874736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r0 = r0 + r6; 2884736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r2 = r2 - r6; 2894736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r6 = r0 - r6 * 2; 2904736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh add r0, r0, r6 2914736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh sub r2, r2, r6 2924736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh sub r6, r0, r6, lsl #1 2934736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 2944736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r5 = r5 * -2.61313 / 8 + r8 = r5 * -21407 / 65536 + r8; 2954736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r8 = r7 * -1.08239 / 8 + r8 = r7 * -8867 / 65536 + r8; 2964736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh smlawt r5, r5, r11, r8 2974736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh smlawb r8, r7, r11, r8 2984736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 2994736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r4 = r4 + r2; 3004736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r0 = r0 + r3; 3014736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r2 = r4 - r2 * 2; 3024736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh add r4, r4, r2 3034736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh add r0, r0, r3 3044736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh sub r2, r4, r2, lsl #1 3054736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 3064736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r7 = r5 * 8 - r3 = -(r3 - r5 * 8); 3074736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r3 = r0 - r3 * 2; 3084736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r1 = r1 - r7; 3094736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r4 = r4 + r7; 3104736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r5 = r8 * 8 - r1 = -(r1 - r8 * 8); 3114736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r7 = r4 - r7 * 2; 3124736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh rsb r7, r3, r5, lsl #3 3134736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh sub r3, r0, r3, lsl #1 3144736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh sub r1, r1, r7 3154736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh add r4, r4, r7 3164736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh rsb r5, r1, r8, lsl #3 3174736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh sub r7, r4, r7, lsl #1 3184736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 3194736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r2 = r2 + r1; 3204736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r6 = r6 + r5; 3214736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r1 = r2 - r1 * 2; 3224736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // r5 = r6 - r5 * 2; 3234736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh add r2, r2, r1 3244736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh add r6, r6, r5 3254736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh sub r1, r2, r1, lsl #1 3264736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh sub r5, r6, r5, lsl #1 3274736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 3284736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // Step 3: Reorder and Save. 3294736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 3304736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // Load output pointer. 3314736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh ldr r8, [r12], #4 3324736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 3334736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // For little endian: r6, r2, r4, r0, r3, r7, r1, r5. 3344736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh pkhtb r6, r6, r4, asr #16 3354736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh pkhtb r2, r2, r0, asr #16 3364736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh pkhtb r3, r3, r1, asr #16 3374736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh pkhtb r7, r7, r5, asr #16 3384736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh usat16 r6, #8, r6 3394736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh usat16 r2, #8, r2 3404736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh usat16 r3, #8, r3 3414736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh usat16 r7, #8, r7 3424736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh orr r0, r2, r6, lsl #8 3434736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh orr r1, r7, r3, lsl #8 3444736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 3454736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh#ifdef __ARMEB__ 3464736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // Reverse bytes for big endian. 3474736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh rev r0, r0 3484736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh rev r1, r1 3494736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh#endif 3504736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 3514736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh // Use STR instead of STRD to support unaligned access. 3524736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh str r0, [r8, r14] ! 3534736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh str r1, [r8, #4] 3544736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 3554736a38e2514bfe50b0241d2053befea82822944Chia-chi Yehpass2_tail: 3564736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh adds r9, r9, #0x10000000 3574736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh bpl pass2_head 3584736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 3594736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh ldr sp, [sp, #8] 3604736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh add sp, sp, #236 3614736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 3624736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh ldmia sp!, {r4, r5, r6, r7, r8, r9, r10, r11, r12, r14} 3634736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh bx lr 3644736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh .endfunc 3654736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh 3664736a38e2514bfe50b0241d2053befea82822944Chia-chi Yeh#endif 367