Click here for the chart's source.
Notice that Republicans become increasingly conservative in the 95th Congress which was elected in 1976 - the first election following the Watergate scandal, and the election that brought Jimmy Carter to the White House. And also that the ideological drift is more pronounced among Republicans than Democrats.
The story that is commonly told about about the roots of ideological polarization is that it begins with (Texas Democrat) Lyndon Johnson's signature of the 1964 Civil Rights Act. Doing so re-positioned the Democratic party - which had historically supported slavery and racial segregation - to being the party of desegregation and civil rights.
Republicans - notable Richard Nixon in 1968 - saw an opportunity to flip the south from being a Democratic stronghold - to a Republican one. The process for doing so was called the Southern Strategy and it involved a series of positions that indirectly touched on race, but mostly focused on the animosity southerners had towards a national government that was forcing them to change laws related to racial segregation.
The above graph reflects the gradual shift of the South - a conservative area of the country - from the Democratic to Republican party.
Note that these efforts - begin in the 1960s - have a delayed effect in Congress. There are many reasons for this, most having to do with the fact that Democrats dominated Congress until the election of 1994. They were the majority party for 40 uninterrupted years. Being a member of the majority party allows access to goodies that are then beneficial to the local constituencies.
But this would not happen for some time - so why do we see the shift occur when it does?