The Hamilton County Sheriff's Office took Ward into custody and arrested him Thursday during a traffic stop in Cincinnati.
Investigators said they determined Ward had a relationship with Kramer from December 2016 until it ended abruptly in May 2017.
"Ward believed Kramer was responsible for destroying him," Boone County sheriff's officials wrote in a news release.
They have said they not think the child was a target. He may have been killed along with his mother so he couldn't testify.
Authorities said got the break in the case they needed to charge Ward from a Crime Stoppers tip.
He did some target practice on a farm, and shell casings found there matched the murder weapon, Major Tom Scheben announced Thursday.
Double murder suspect Joshua Ward waives extradition hearing in court this morning and will return to NoKY for the shooting death of ex girlfriend Kelli Kramer and her 9 year old son. More on @FOX19pic.twitter.com/cI5BZihfNi