The Japanese Navy had initiated a crash course in attempting to create a world-class fleet, starting from scratch in the late 19th c., even copying the boiler patches on western ships. The arms race in the nineteen twenties and thirties left them far behind the American Navy in doctrine and tactics at the outset of WWII. It wasn't a level playing field.
They had okay ships and planes; their failure was a lack of tradition in handling their new toys.
John Prados’s book Islands of Destiny, about the Solomons Campaign, reinforces this impression (there may be an American bias there I guess). The Americans act decisively – especially after the departure of Fletcher and Ghormley in September 1942. In contrast the Japanese were operating in the dark, never quite sure what enemies they might encounter. A huge intel gap was the key that allowed Americans to act more boldly.
In “Combined Fleet Decoded,” Prados writes:
Contrasting Japanese practice with the proliferation of situation rooms and flag plots maintained by Allied intelligence is easy. “To make a chart showing positions of individual ships at any one time was beyond the competence of the 3rd Bureau.” Admiral Ono said, “We did try to decide which units of your fleet were in general large areas.” That intelligence would be passed to the First Bureau, which relied upon it as being from a good source. Questioned on this point, Baron Tomioka replied that the most useful data from joho kyoku were weekly written estimates of Allied losses, strength, and locations; the oral information presented at the daily staff conference; and character analysis of Allied commanders. On the whole, Tomioka said, intelligence was “very poor, very haphazard.”
Captain Ohmae Toshikazu, Tomioka’s eventual successor as operations planner with Naval General Staff, remarked:
“The 5th section collects all information, checks it, makes their evaluation, throws out information which is unreliable. It takes the 5th Section a long time to make an evaluation of most of the information. They then send their opinion to us. We also check it, and it may be a few more days before we are satisfied enough to use it. There were some difficulties.”
When the Japanese Eighth Fleet sailed to Guadalcanal and crushed the American Cruiser force providing cover for the transports unloading at Lunga Point, Admiral Mikawa made the decision not to turn around to sink the American transports because he was unaware of the location and heading of American carriers.
Aerial reconnaissance did, however, provide Mikawa an accurate picture of the transport fleet. Around 1200 on August 8th a Japanese scout plane which had been reconnoitering the Tulagi-Guadalcanal area was recovered and Mikawa reformed. The pilot gave a most detailed account of the Allied forces in the Tulagi-Guadalcanal area:
(a) In the Lunga Point area, one battleship, four cruisers and seven destroyers, one ship resembling an escort carrier, and fifteen transports. He also reported that the cruisers and destroyers were conducting a roving patrol and that the transports were at anchor.
(b) In the Tulagi area, two heavy cruisers, twelve destroyers and three transports all underway in the vicinity of the Tulagi lighthouse.
The above intelligence was surprisingly accurate. There were, of course, neither the battleship nor escort carrier present. There were eight cruisers rather than six; twenty-four destroyers, destroyer transports and minesweepers rather than nineteen; and nineteen large transports or supply ships rather than eighteen. The Allied surface forces were caught unaware and routed, losing one Australian and three American cruisers. Allied dead totaled 1,023; 709 personnel were wounded. The Japanese only suffered light damage. The battle has come to be identified as the worst defeat in a single fleet action suffered by the United States Navy.
Mikawa's decision to withdraw under cover of night rather than attempt to destroy the Allied invasion transports was primarily founded on concern over possible Allied carrier strikes against his fleet in daylight. In reality, the Allied carrier fleet, similarly fearing Japanese attack, had already withdrawn beyond operational range. Mikawa’s withdrawal led to the Japanese failure on Guadalcanal. If most of the transport fleet had been destroyed, the Americans would have been the ones to starve on “Starvation Island.” The Guadalcanal Campaign was as good as lost at the opening of the campaign, because of a lack of intel.
After the war, Ohmae wrote: “Based on radio intelligence of the previous evening, we knew that there were enemy carriers about 100 miles southeast of Guadalcanal. As a result of our night action these would be moving toward the island by this time, and to remain in the area by sunrise would mean that we would meet the fate our carriers had suffered at Midway.”
“The withdrawal of the Eighth Fleet without having destroyed the enemy transports has, since that time, come in for bitter criticism, especially after it was disclosed that the enemy carrier force was not within range to attack and, most especially, after our army was unable to dislodge the enemy from Guadalcanal. It is easy to say, now, that the enemy transports should have been attacked at all cost. The validity of this assumption, however, is premised on the fact that the survival of those transports accounted entirely for our army’s subsequent failure to expel the enemy from its foothold in the Solomons.”
It seems that the Japanese never realized to what degree all their plans were compromised because of the “American Magic.” Time and again their initiatives were stopped cold. Even when American code breakers identified Admiral Yamamoto’s flight plans and his plane was shot down in April of 1943, almost no one seriously wondered whether the Americans were reading their codes.
“Commander Watanabe Yasuji understandably obsessed over everything that preceded Yamamoto’s death. He discovered that his original messages (with the Admiral’s itinerary), supposed to have been confined to Navy radios, had in fact passed over Army circuits also. Tis bit of confusion created an obstacle to the Imperial Navy’s realizing its codes had been cracked.” Prados, Islands, p. 274
And yet, when pressed, the Japanese Fleet was able to outwit the Americans. During the KA Operation to evacuate Japanese troops from Guadalcanal in February of 1943, the Japanese Kido Butai set sail as if to presage another offensive. This caused Admiral Nimitz to withdraw his cruiser force from the Slot and gave the IJN time to successfully evacuate over 10,000 half-starved troops over a period of three days.
In 1944 the senior air commander at Truk was Rear Admiral Hasegawa Kiiichi of the 22nd Air Flotilla. “Hasegawa came as close as anyone to unraveling the truth” that the D Code (JN-25) had been broken. “The suspicion was based on the way a tanker had been sunk right outside the entrance to Kwajalein after the Japanese had sent several messages related to her route.” The tanker’s destination was made clear by the headings on the dispatches. But before Hasegawa could report his findings an American carrier raid on Truk killed him. Prados, Combined Fleet, p. 536