The biggest downsides of using FAT32 on large drives are:
1. The 4GB filesize limit.
2. Cluster size/FAT size. The smaller your clusters, the bigger your FAT, and vice versa. If you're going to have a large number of small files, go for a smaller cluster size. If it's going to be mostly large files (tens or hundreds of MBs apiece), go for a larger cluster size.
Elaborating on point 2 a bit, if you go with 8KB clusters on a 1TB drive, you'll be wasting about half a gig on the file allocation table itself. Double the cluster size and the FAT shrinks by 50% (256MB). Go up to 32KB clusters and and your FAT is only 128MB, but it also means a tiny text file with a few lines will gobble up 32KB of disk space.
There's also no data integrity protection on FAT32. The main advantages of FAT are its performance in situations where a disk is not heavily fragmented, you have mostly large files, you don't keep a large number of files (1000+) in a single directory, and you have large cluster sizes. Journaled file systems like ext2/3/4 are slower per operation but data integrity is guaranteed. They also have much less slack (wasted) space, if I remember correctly, because they can use very small clusters.
In essence, whether or not you should use FAT32 will depend on your specific needs. Personally, I never see a good reason to use FAT32 when NTFS or a flavor of ext will do.