KUALA LUMPUR, Malaysia — A challenge to a law barring Muslim men from dressing or posing as women was dismissed on Thursday by a Malaysian court, prompting concern that more transgender people in the Muslim-majority country may be prosecuted.