The description of the Solitaire analysis dialog is here: Solitaire analysis dialog.
For the Solitaire cipher, there is kind of an attack (ciphertext-only attack), which makes a automatical or semi-automatical key search. For that it is necessary to know the final deck and the related ciphertext.
Because the algorithm is strong reversible, by reversing the steps of the algorithm the initial deck can be generated and so the symmetric key. With this key the ciphertext can be decrypted. The problem of the analysis is the non-complete reversibility of the algorithm with the effect that not each step can be reversed in an unique manner. If a wrong way is chosen, the further analysis is also wrong.
The analysis has three general steps:
1. Determining the initial deck
2. Generating the keystream
3. Decryption of the ciphertext
1. Determining the initial deck
The Solitaire algorithm is passed through backwards for each ciphertext letter. That means it starts with step 5, then follows step 4 and so on.
Step 5:
No letter is noted. It is just proofed, if there is a zero round. In this case the algorithm has to be passed through backwards once more.
Step 4:
The cards, which have been sorted in front of the last card at step 4, have to be sorted in at the top of the deck. The number of cards is represented by the card at the bottom.
Step 3:
Swap the stack of cards in front of the first joker with the stack of cards behind the second joker.
Step 2:
Joker B is sorted in two cards before its current position.
Step 1:
Joker A is sorted in one card before its current position.
After the algorithm is passed through backwards for each ciphertext letter and for each zero round, the initial deck is generated. Because of the aspect, that step 1 and step 2 cannot reversed in an unique manner, the algorithm is not completely reversible. If one of the jokers is sorted in at the wrong position, the decryption leads to a wrong result.
It follows the same procedure as at the decryption with Solitaire.
2. Generating the keystream
Now, the algorithm is used once on the initial deck for each ciphertext letter and for each zero round (Solitaire keystream generation ). So the keystream, which has already been used for the encryption of the plaintext is created.
3. Decryption of the ciphertext
The ciphertext will be decrypted with this keystream.