A "challenge" is a cryptographic problem that has been published, sometimes with a prize offered for cracking it.
When a challenge has not yet been cracked despite many attempts, then confidence in the propagated system rises. If someone succeeds in cracking the problem, then the cryptosystem with the specified parameters is no longer regarded as secure.
Challenges are regularly published on the RSA website (www.rsa.com) for the best-known cryptosystem, the RSA system of Rivest, Shamir and Adleman.