Thank you for reply! So considering your remark, lets recount it: 15FC0D48(word0) D7F8199C(word1) BE399183(word2) 4D96F327(word3) 10000000(word4) 00000000(word5) 00000000(word6) 00000000(word7). Afterwards wi=(wi−8⊕wi−5⊕wi−3⊕wi−1⊕ϕ⊕(i−8))⋘11 word8 8-15 i=8 15FC0D48 xor 4D96F327 xor 9E3779B9 <<<11=EC3EB632 next w0=w8 As you have told this formula is applied for 8-131 wi=(wi−8⊕wi−5⊕wi−3⊕wi−1⊕ϕ⊕i)⋘11 but EC3EB632 is not equals to afb59841 which is k0{w0}

My values are AFTER the application of the sboxes (final subkeys); before the sbox the value w(0) is EC3EB632, which will be 32b63eec in my print format (big endian). Then ffb0b08e 753dc6f2 6ad60ec8 for w(1-3)

EC|3E|B6|32 W0 32B63EEC W0 in BigEndian, converting it to binary to Apply Sbox3 Transformation 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 00/11/00/10/10/11/01/10/00/11/11/10/11/10/11/00 00001000111101101011001011101111 8F6B2EF finally it is w0, it not matches in your table with 00 = afb59841. Could you please say is anything wrong? So Serpent uses BigEndian values for all operations? Or BigEndian/LittleEndian but converting ciphertext back to BigEndian. Can I contact you with e-mail, I also have my implementation of Serpent but I never read that I should apply Little/BigEndian.

Your implementation is also can transfer plaintext to cipher and than back to plaintext? For Sbox Transformation I have wrote this function String key="10011100000011101101011001101011"; prek=""; Uint32 k; for (int i = sboxpos; i < sboxpos + 1; i++) { for (int i2 = 0; i2 < 16; i2++) { prek += key.Substring(sbox[i, i2] * 2, 2); //Multiplying on 2 to move byte on appropriate position, as in my example above I transform hex to binary it have 32bits, but Sbox'es moves only bytes. So e.g. the 15's byte is 15*2=30's position plus 2 characters } } k = Convert.ToUInt32(prek, 2);

Remember bit-sliced, so you will need the permutation to reorder the bits once the entire 128-bit subkey group has been sent through the s-box to match my values. I think if you run my subkeys through the final permutation you will get what you expect for a manual calculation. Also my key input is also big endian, treat 15FC0D48.. as a byte stream. I will update the answer with w() values

Thanks! I assume that I do something wrong in Sbox. I have reviewed official submission package of Serpent and noticed that in Sbox they use AND(&) OR(|) operands. Here is how I understand Sbox transform:Lets have this HEX B0:EC3EB632 In binary(I mark them as bytes 0-15, 16*2=32bytes: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 11|10|11|00|00|11|11|10|10|11|01|10|00|11|00|10| After Sbox3 Appliance (byte 0 goes on 0 position (11), byte 15 goes on 1st position(10), byte 11 goes on 2nd position(10) and so on until I reach byte 14: 11101010001111001110110001101100 EA3CEC6C Are you doing same?

No, that is not how an sbox works. it takes a 4-bit input and gives a 4-bit output, S3(5) = 9

EC3EB632 run through S3 becomes 5A85468B

it is not just big endian

it is bitsliced

the permutation interleaves the bits between all 4 words

But I also noticed that sometimes there is a situations when HEX gives non full 32bits binary characters. Can I just add '0''s in front of binary to fill it up to 32bits length?

yes

And my last question, since I disturbed you so much. I have recounted the first w0-w15 placed it to w0-w7 and applied S3 at K0{w0} but it did not matched with your table 'Working register values w()' and 'Subkeys k() after application of s-boxes (bit-sliced)' above. I will not use bit sliced mode, but will store initial 256bit key in Uint32 Array of size of 8 and plain text in Uint32 Array of size of 4.

Will I be able at lease decypher 50% of ciphertext to plaintext if I will compute all operations as is with out any Bigendian/Littleendian? Thank you again for such precise explanation of question!

as i said, mine is bitscliced

Ok, Can I post my code here when I will finish it? I will use C#

sure, but if you are doing this the way you currently are, it will never match my results

my s-boxes and linear transformations are bitsliced

And I will not able to decipher initial plaintext?

they operate on 32 4-bit values at the same time

plaintext/ciphertext will still match

but the intermediate round values and round keys will not

since the bitsliced version does not require the initial or final permutation

Firstly I press Button1Function which computes RoundKey, than after I have computed RoundKeys I launch encryption() function which does 32-1 rounds (final round) 32 not uses Linear Transformation only layering additional K31/K32. Than I get my Ciphertext, after that I press Button2 Function which does inverse operations of encryption and finally I get my plaintext back.

But considering my test key 15FC0D48 D7F8199C BE399183 4D96F327 10000000 00000000 00000000 00000000 and plainText 00000000 00000000 00000000 00000000 And after that final decrypted text is not 00000000 00000000 00000000 00000000

I not used any Big/Little Endian all computations are same if I have did it manually on paper. For convinience I placed my key in seperate UINT32 variables to avoid slicing and I have placed my plaintext at UINT32[4] Array.