Its fixed. I ended up loading all 64 bytes of the attribute table as soon the first part of the next nametable started to be drawn. I then had to end my drawing after 960 bytes were drawn instead of 1024 so I didn't draw the attributes twice and screw up my timing.
It was really tricky to get this working, I ended up creating a subroutine which put the location of the correct attribute table in addresses $12 and $13, then I could work with that in the NMI. It is awesome to be able to walk down 10 screens worth of data.
Now I will start sprite 0 detect so my dude doesn't walk right over mountains.