Which group within the federal government has the authority to limit the president's inherent powers?

Congress--Congress and the President are the only two branches with inherent powers and they are able to check each other in the process.

Inherent powers are not outlined in the Constitution but are the powers Congress and the President have to do their job right. These come through interpretation of the Constitution and the power of the positions.