The Amnesty report said Nigeria's security forces acted outside the rule of law and their brutal tactics could build support for Boko Haram outside its extremist core.
A Nigerian military spokesman contacted by Reuters rejected the report as "biased and mischievous".
"The cycle of attack and counter-attack has been marked by unlawful violence on both sides, with devastating consequences for the human rights of those trapped in the middle," said Salil Shetty, Secretary General of Amnesty International.
"Every injustice carried out in the name of security only fuels more terrorism, creating a vicious circle of murder and destruction."
The report is likely to add to calls for Nigeria's military to change its heavy-handed approach to tackling the insurgency, which critics have long said is driving desperate youths into the arms of Boko Haram.
It details cases of abuses stretching back to the start of the Boko Haram uprising in 2009.
The report said a "significant number" of people accused of links with Boko Haram had been executed after arrest without due process, while hundreds were detained without charge or trial and many of those arrested disappeared or were later found dead.
"People are living in a climate of fear and insecurity, vulnerable to attack from Boko Haram and facing human rights violations at the hands of the very state security forces which should be protecting them," Shetty said.
Amnesty said it had spoken to witnesses who described seeing people who were unarmed and lying down with their hands over their heads shot at close range by soldiers.
In one case, a widow described how soldiers put a gun against her husband's head three times and told him to say his last prayers before shooting him dead. They then burned down their home. She now fends for her seven children alone.
Defense spokesman Colonel Mohammed Yerima said that Nigerian forces only kill Boko Haram suspects during gunfights, never in executions "We don't torture people. We interview a suspect, if he is not involved we let him go. If he is involved we hand him to the police," he said. "I totally disagree with this report. It is biased and it is mischievous."
Amnesty said it had sent a delegation to the states worst affected by the insurgency, Kano and Borno, between February and July to investigate reported atrocities.