How was madison a strict constitutionalist during his presidency? I know mostly he was a Federalist, but there had to be sometime he was not.

Interestingly not. Madison was a huge proponent of virtually the entire constitution. he led the constitutional convention even going so far as arriving to the location early as to prepare his notes. in fact he vetoed many bills on the grounds that they were not constitutional. (bonus bill of 1817) however he is not without fault he is largely responsible for the two party system we still have today however that is not solely his fault