Iwamatsu was born September 21, 1908, in Nejime, Kimotsuki District, Kagoshima, and raised there on the southern coast of Kyushu. His father was a country doctor who collected oriental art and encouraged art in his son. After studying for three years at the Imperial Art Academy in Tokyo, Iwamatsu was expelled for insubordination and missing a military drill. He then joined a group of progressive artists, sympathetic to the struggles of ordinary workers and opposed to the rise of Japanese militarism. The antimilitarist movement in Japan was highly active at the time within many professional groups and crafts, and artists' posters protesting the Japanese aggression in China were widespread. Following the Japanese invasion of Manchuria, however, the Japanese government began heavy handed suppression of domestic dissent including the use of arrests and torture by the Tokkō (Special Higher Police).
Both Iwamatsu and his pregnant wife, Tomoe, were imprisoned and brutalized for their opposition to the militaristic government. In 1939, they left Japan for the United States so Iwamatsu could avoid conscription into the Japanese Army and to study art, leaving behind their son Mako (born 1933). After Pearl Harbor, Iwamatsu joined the U.S. Army and went to work as an artist for the United States Office of War Information (OWI) and, later, the Office of Strategic Services (OSS). It was then that he first used the pseudonym Taro Yashima, out of fear there would be repercussions for Mako and other family members if the Japanese government knew of his employment. After the war, he and his wife were granted permanent resident status by an act of the U.S. Congress. Soon after they had their second child, Momo, while living in New York City. Iwamatsu was able to return to Japan and collect Mako in 1949.